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Supervisory  Control  of  Multiple  Uninhabited 
Systems  -  Methodologies  and  Enabling 
Human-Robot  Interface  Technologies 

(RT  O-TR-HFM- 170) 

Executive  Summary 

Uninhabited  Vehicles  (UVs)  are  at  the  forefront  of  current  operations  and  future  thinking.  With  increasingly 
automated  UVs,  the  operator’s  role  will  become  more  supervisory  in  nature,  overseeing  the  automated 
activation  of  planned  events  and  managing  unexpected  changes  that  impact  the  automated  mission  plans. 
Future  vehicles  will  also  have  the  capability  to  make  certain  decisions  independent  of  operator  input  and 
pre-defmed  mission  plans.  This  capability  of  the  UVs  to  ‘decide’  (i.e.,  be  autonomous)  constitutes  a  whole 
new  set  of  challenges  for  UV  operators,  as  they  will  be  required  to  rapidly  judge  the  appropriateness  of 
certain  UV  decisions  and  assess  their  impact  on  overall  mission  objectives,  priorities,  rules  of  engagement, 
etc.  Moreover,  there  is  a  vision  for  a  new  control  paradigm  whereby  a  single  operator  will  simultaneously 
supervise  multiple  autonomous  UVs.  Unfortunately,  there  is  a  dearth  of  information  as  to  how  best  to 
support  the  coupling  of  this  intelligent  autonomy  with  the  unique  capabilities  and  decision-making 
responsibilities  of  the  operator  so  as  to  maximize  mission  effectiveness  across  a  wide  range  of  mission 
contexts.  New  interfaces  are  required  that  take  into  account  issues  associated  with  this  automation 
management  as  well  as  potential  negative  automation-induced  impacts  on  the  operator. 

Given  the  possibility  that  future  operators  may  likely  control  many  UVs  simultaneously,  additional  human 
factors  challenges  will  include  how  best  to  maintain  situation  awareness,  a  reasonable  workload  level, 
and  high  system  performance  and  safety  across  several  managed  assets.  New  principles  for  supporting  the 
operator  in  such  scenarios,  which  focus  on  supervisory  control  design  methodologies,  adaptive 
automation,  and  novel  situation  assessment/decision  support  aids,  need  to  be  developed  and  evaluated. 
Additionally,  standard  operator  interface  design  guidelines  associated  with  UV  supervisory  control  need  to 
be  identified  so  as  to  facilitate  interoperability  across  unmanned  platforms. 

HFM-170  developed  and  demonstrated  pertinent  supervisory  control  human-system  interface  design 
practices  and  concepts  for  UV  network-centric  operations  through  15  specific  technology  demonstrations. 
These  demonstrations  focused  on  many  critical  issues  including  multi-vehicle  control,  manned-unmanned 
teaming,  human-automation  interaction,  telepresence  interfaces,  delegation  interfaces,  vehicle  hand-offs, 
operator  workload  adaptive  systems,  variable  levels  of  autonomy,  authority  sharing,  situation  awareness 
aids,  cognitive  workload  assessment,  swarming  interfaces,  and  dynamic  mission  management.  HFM-170 
concentrated  on  the  identification  and  demonstration  of  successful  supervisory  control  methodologies  and 
interface  design  practices  for  enabling  single  operator  control  of  multiple  UVs.  The  applications  addressed 
varied  in  degree  of  autonomy  from  manual  robotic  control  to  highly  autonomous,  swarming  UVs. 

This  report  summarizes  in  alphabetical  order  these  15  Technology  Demonstrations,  including  a  summary 
description  of  the  activity,  human  factors  issues  involved,  results  and  lessons  learned.  This  report  also 
provides  a  discussion  on  the  development  of  a  supervisory  control  framework  by  which  to  characterize 
and  communicate  research  and  technology  development  occurring  within  the  supervisory  control  domain. 
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Commande  et  surveillance  de  multiples  systemes  sans 
pilote  -  Methodologies  et  technologies  habilitantes 
d’interfaces  homme-machine 

(RT  O-TR-HFM- 170) 

Synthese 

Les  vehicules  sans  pilote  (UV,  Uninhabited  Vehicle )  occupent  une  place  de  premier  plan  dans  les  operations 
actuelles  et  dans  les  etudes  de  prospective.  Avec  la  generalisation  des  vehicules  automatises  sans  pilote, 
l’operateur  aura  de  plus  en  plus  un  role  de  surveillance,  qui  consistera  a  superviser  1’ activation  automatisee 
d’evenements  planifies  et  a  gerer  les  changements  imprevus  susceptibles  d’ avoir  une  incidence  sur  les  plans 
de  mission  generes  automatiquement.  Les  vehicules  futurs  seront  egalement  capables  de  prendre  certaines 
decisions  independamment  de  toute  intervention  de  l’operateur  et  du  plan  de  mission  predefini.  Cette 
capacite  de  decision  (autrement  dit,  cette  autonomie)  des  vehicules  sans  pilote  pose  des  defis  d’un  type 
entierement  nouveau  aux  operateurs  de  vehicules  sans  pilote  :  ils  seront,  en  effet,  appeles  a  evaluer  rapidement 
1’ adequation  de  certaines  decisions  prises  au  niveau  du  vehicule  sans  pilote  et  a  en  apprecier  1’ impact  sur  les 
objectifs  generaux  de  la  mission,  les  priorites,  les  regies  d’engagement,  etc.  Qui  plus  est,  un  nouveau 
paradigme  de  commande  emerge,  dans  lequel  un  operateur  unique  aura  a  surveiller  simultanement  de 
multiples  vehicules  sans  pilote  et  autonomes.  Malheureusement,  on  manque  d’ informations  sur  les  meilleurs 
moyens  d’ assurer  le  couplage  entre  cette  autonomie  intelligente  et  les  capacites  et  responsabilites 
decisionnelles  uniques  de  l’operateur  en  vue  d’optimiser  l’efficacite  de  la  mission  dans  une  grande  diversite 
de  contextes  de  mission.  De  nouvelles  interfaces  sont  necessaires  pour  prendre  en  compte  les  questions  liees 
a  cette  gestion  de  l’automatisation  et  les  effets  negatifs  sur  1’ operateur  potentiellement  induits  par 
l’automatisation. 

Etant  donne  que  les  futurs  operateurs  risquent  de  devoir  controler  simultanement  un  grand  nombre  de 
vehicules  sans  pilote,  d’autres  problemes  relatifs  aux  facteurs  humains  se  posent,  en  particular  la  meilleure 
fagon  de  garantir  une  connaissance  adaptee  de  la  situation,  une  charge  de  travail  raisonnable,  et  un  niveau 
eleve  de  securite  et  de  performance  du  systeme  sur  un  ensemble  gerant  plusieurs  engins.  De  nouveaux 
principes  pour  soutenir  l’operateur  dans  de  tels  scenarios,  axes  sur  les  methodologies  de  conception  de 
systemes  de  commande  avec  dispositif  de  surveillance,  l’automatisation  adaptative  et  sur  les  aides  a 
1’evaluation  d’une  situation  nouvelle  et  a  la  decision,  doivent  etre  developpes  et  evalues.  De  plus,  des  lignes 
directrices  pour  la  conception  d’interfaces  operateurs  standard  associees  a  un  systeme  de  commande  avec 
dispositif  de  surveillance  de  vehicules  sans  pilote  doivent  etre  definies  de  maniere  a  faciliter  l’interoperabilite 
entre  des  plates-formes  non  habitees. 

HFM-170  a  mis  au  point  et  demontre  des  pratiques  pertinentes  pour  la  conception  d’interfaces  homme- 
machine  avec  dispositif  de  surveillance  et  des  concepts  pour  des  vehicules  sans  pilote  operant  dans  un 
contexte  d’operations  reseaucentriques  dans  le  cadre  de  15  demonstrations  des  technologies  specifiques. 
Ces  demonstrations  ont  mis  1’ accent  sur  plusieurs  points  cruciaux,  parmi  lesquels  la  commande  de  vehicules 
multiples,  le  travail  d’equipe  homme-machine,  l’interaction  homme-automatisation,  les  interfaces  de 
telepresence,  les  interfaces  de  delegation,  les  vehicules  automatises  sans  intervention  manuelle  (i hand-offs ), 
les  systemes  adaptatifs  de  charge  de  travail  de  l’operateur,  les  niveaux  variables  d’autonomie,  le  partage 
d’autorite,  les  aides  a  la  connaissance  de  la  situation,  1’evaluation  de  la  charge  de  travail  cognitive, 
les  interfaces  de  systemes  en  essaim  {swarming),  et  la  gestion  dynamique  de  missions.  HFM-170  a  concentre 
son  attention  sur  1’ identification  et  la  demonstration  de  methodologies  de  commande  avec  dispositif  de 
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surveillance  et  de  pratiques  de  conception  d’ interfaces  efficaces  pour  permettre  a  un  operateur  unique  de 
commander  de  multiples  vehicules  sans  pilote.  Les  applications  concemees  correspondaient  a  un  niveau 
d’autonomie  variable  allant  de  la  commande  robotisee  manuelle  a  des  vehicules  sans  pilote  tres  autonomes  et 
operant  en  essaim. 

Ce  rapport  presente  un  resume  par  ordre  alphabetique  de  ces  15  Demonstrations  des  technologies, 
notamment  une  description  sommaire  de  l’activite,  les  questions  relatives  aux  facteurs  humains  s’y  rapportant, 
les  resultats  obtenus  et  les  enseignements  qui  en  sont  tires.  Le  rapport  discute  egalement  de  1’evolution 
d’un  cadre  de  commande  avec  dispositif  de  surveillance  adapte  pour  caracteriser  revolution  des 
recherches  et  des  technologies  dans  le  domaine  du  controle  et  de  la  surveillance  et  en  assurer  la  diffusion. 
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Chapter  1  -  INTRODUCTION 
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1.1  OVERVIEW  AND  SCOPE 

Uninhabited  Vehicle  systems  (UVs)  are  at  the  forefront  of  current  battles  and  future  thinking.  A  number  of 
NATO  countries  are  now  using  UVs  to  enhance  their  manned  forces,  especially  in  performing  tasks  that  are 
dull,  dirty,  or  dangerous.  While  several  projects  are  focused  on  increasing  the  level  of  autonomy  for  future 
UVs  (and  thus  enabling  supervisory  control),  there  is  a  dearth  of  information  as  to  how  best  to  couple  this 
intelligent  autonomy  with  human  decision-making  abilities.  With  highly  automated  UVs,  the  operator’s  role  is 
supervisory  in  nature,  overseeing  the  automated  activation  of  programmed  events  (e.g.,  making  sure  the 
appropriate  event  is  activated  at  the  appropriate  time)  and  managing  unexpected  changes  to  the  automated 
mission  plan.  Associated  operator  interfaces  must  take  into  account  issues  associated  with  automation 
management,  including  vigilance,  attention  management,  clumsy/brittle  automation,  etc.  Continuing  this  trend 
beyond  the  current  state-of-the-art,  a  vision  exists  for  a  new  interface  paradigm  for  controlling  next-generation 
UVs.  This  envisioned  interface  system  involves  multiple  autonomous  UVs  being  controlled  by  a  single 
supervisor.  These  UVs  will  have  the  capability  to  make  certain  decisions  independent  of  operator  input  and 
pre-de fined  mission  plans.  This  capability  of  the  UV  to  ‘decide’  constitutes  a  whole  new  set  of  challenges  for 
UV  operators,  as  they  will  be  required  to  rapidly  judge  the  appropriateness  of  these  decisions  and  assess  their 
impact  on  overall  mission  objectives,  priorities,  etc. 

Given  the  current  progress  of  technological  developments  and  operational  concepts  regarding  UVs,  a  strong 
and  combined  effort  of  NATO-countries  is  essential  to  resolve  the  unique  human-system  issues  associated 
with  augmenting  the  existing  force  with  these  vehicles.  Since  the  trend  is  very  clearly  on  the  development  of 
more  autonomous  UVs,  the  time  is  right  to  address  the  critical  human  factors  issues  involved.  Human  factors 
design  guidelines  will  have  the  greatest  impact  if  they  are  identified  before  wide  scale  NATO  design  and 
procurement  of  highly  autonomous  UVs  occur.  Given  the  possibility  that  future  operators  may  control 
multiple  UVs  simultaneously,  additional  human  factors  challenges  will  be  to  maintain  situation  awareness, 
a  reasonable  workload  level,  and  high  system  performance  and  safety  across  several  managed  assets. 
New  principles  for  supporting  the  operator  in  such  scenarios,  which  focus  on  supervisory  control  design 
methodologies  and  novel  situation  assessment/decision  support  aids,  need  to  be  developed  and  evaluated. 
Additionally,  standard  operator  interface  design  guidelines  associated  with  UV  supervisory  control  need  to  be 
identified  so  as  to  facilitate  interoperability  across  unmanned  platforms.  The  ultimate  goal  of  HFM-170  was  to 
increase  NATO’s  successful  operations  utilizing  highly  automated  UVs;  however,  the  specific  goal  was  to 
provide  a  single  point  of  focus  for  identifying,  prioritizing,  and  addressing  human  factors  challenges 
associated  with  UV  supervisory  control. 

HFM-170  team  members  developed  and  demonstrated  pertinent  supervisory  control  human-system  interface 
design  practices  and  concepts  for  UV  network-centric  operations.  It  directly  leveraged  HFM  Task  Group 
HFM-078/RTG-017  [1],  which  developed  a  comprehensive  review  of  uninhabited  military  vehicle  human 
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factors  issues  across  a  wide  variety  of  human  effectiveness  areas  and  potential  military  applications.  Building 
off  this  acquired  knowledge,  HFM-170  concentrated  on  the  identification  and  demonstration  of  successful 
supervisory  control  methodologies  and  interface  design  practices  for  enabled  single  operator  control  of 
multiple  UVs,  with  various  degrees  of  autonomy  (including  highly  autonomous  UVs). 

Several  relevant  issues  and  challenges  addressed  included: 

•  Supervisory  Control  Issues  and  Methodologies: 

•  Human-automation  challenges  and  mitigation  techniques. 

•  Human-automation  problem  solving/cooperative  dialog. 

•  Networked  telepresence. 

•  Manned/unmanned  collaboration. 

•  Flexible  (adaptive)  level  of  automation. 

•  Optimization  of  human/vehicle  ratio. 

•  Heterogeneous  systems. 

•  Control  Station  Design  -  Decision  Support  Interfaces: 

•  Situation  assessment  aids,  augmented  feedback  of  action  impact. 

•  Task  switching,  interruption  and  prioritization  methods. 

•  Predictive  /  “look  ahead”  tools,  anticipatory  support. 

•  Intelligent  aiding,  time-critical  decision  making. 

•  Multi-modal  interfaces,  intuitive  interfaces,  natural  language  speech  enabled  interfaces. 

•  Commonality  of  supervisory  control  interface  design  components  supporting  interoperability. 

•  Augmented  remote  world. 

A  unique  aspect  of  HFM-170  was  the  process  followed.  The  team  was  given  explicit  instruction  to  operate  in 
a  more  collaborative  manner,  with  more  demonstrations  versus  discussions  of  research  papers.  The  next 
section  discusses  a  novel  approach  that  the  group  settled  on  to  attempt  to  maximize  collaboration  and  tech 
demos  without  compromising  each  researcher’s  research  priorities. 


1.2  HFM-170  PROCESS:  MAXIMIZE  COLLABORATIONS  AND  TECHNOLOGY 
DEMONSTRATIONS 

Given  the  direction  from  the  HFM  Panel  for  the  Task  Group  to  focus  on  increasing  team  collaborative  efforts 
and  hosting  high-fidelity  Technology  Demonstrations  (TDs)  versus  strictly  discussing  lab  research  findings, 
the  team  needed  to  formulate  a  new  approach  to  facilitate  these  objectives.  However,  the  dilemma  was  how  to 
accomplish  true  collaboration  within  the  obvious  limitations  that  exist  with  NATO  teams  (e.g.,  no  additional 
resources  provided,  conflicting  schedules,  international  restrictions,  the  continuing  need  for  team  members  to 
accomplish  their  own  national  research  agenda).  HFM-170  Team  Members  thus  formulated  a  new  process  by 
which  to  formally  identify,  develop  and  ascertain  NATO  collaboration  potential  for  specific  UV-related  TDs 
that  would  be  occurring  within  each  individual  country  over  the  time-course  of  the  Task  Group.  This  process 
is  summarized  below. 
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The  team  first  identified  a  series  of  TDs  that  would  occur  throughout  the  follow-on  Task  Group  period  of 
performance.  Each  participating  Nation  was  allotted  at  least  one  TD  if  they  so  desired.  A  total  of  15  Technology 
Demonstrations  (TDs)  were  eventually  agreed  upon  across  8  countries.  These  TDs  focused  on  a  broad  range  of 
pertinent  human  factors  issues  associated  with  supervisory  control  of  multiple  unmanned  systems  (see  next 
section  and  the  following  chapters).  Several  candidate  supervisory  control  frameworks  were  subsequently 
conceived  in  an  effort  to  integrate  these  TDs  into  a  common  supervisory  control  framework  (see  Chapter  2). 

After  identification  of  the  official  list  of  TDS,  each  TD  was  considered  in-tum  for  potential  level  of  NATO 
collaboration.  Since  higher  levels  of  international  collaboration  requires  a  significant  amount  of  lead  time  for 
planning  and  orchestrating,  this  discussion  of  potential  collaboration  opportunities  took  place  at  the  initiation  of 
the  Task  Group.  Collaboration  among  each  of  the  participating  TG  NATO  Nations  was  considered  along  a 
graduated  scale  (Figure  1-1).  This  scale  defaults  at  ‘no  collaboration’,  which  is  applicable  to  many  situations 
given  constraints  placed  on  programs  and  costs  of  collaboration.  As  collaboration  level  increases,  the  scale  rises 
to  “coordination”  (information  sharing,  schedule  coordinating,  witnessing  the  TD,  etc.),  then  to  “cooperation” 
(structuring  similarly  focused  tech  demos  to  enhance  effects,  maximize  information  gathering,  data  collection) 
and  finally  to  full  “collaboration”  (multiple  NATO  Nations  combine  resources  to  produce  a  truly  integrated  TD). 
Full  collaboration  was  achieved  in  one  instance  within  this  Task  Group,  and  is  described  in  Chapter  9. 
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Figure  1-1:  Levels  of  NATO  Technology  Demonstration  Collaboration  for  HFM-170. 
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For  each  TD,  the  eventual  level  of  collaboration  for  each  country/representative  was  dependent  upon  several 
variables  including  level  of  mutual  research  interest,  availability  of  resources,  alignment  of  resources,  timing, 
value  added,  etc.  The  method  for  identifying  and  characterizing  instances  of  collaboration  is  described  next. 

Each  TD  “owner”  was  required  to  complete  a  collaboration  matrix  (see  Figure  1-2  below)  that  conveyed  how 
much  collaboration  was  desired  (and  in  what  area  of  the  TD).  One  dimension  of  the  matrix  consisted  of 
3  levels  of  collaboration  while  the  other  represented  3  different  phases  of  a  TD.  For  each  TD,  this  completed 
matrix  was  presented  along  with  a  discussion  of  the  TD  (objectives,  approach,  design,  etc.),  after  which  each 
country  was  prompted  to  state  their  level  of  interest  (using  the  same  collaboration  matrix  structure)  in 
collaborating  with  that  TD.  In  this  way,  the  group  was  able  to  systematically  identify  and  then  track  collaborations 
across  a  wide  spectrum  of  collaboration  levels  and  groups  involved.  Some  TDs  resulted  in  few  to  no 
collaborations  while  other  TDs  had  much  interest  from  various  countries  and  one  resulted  in  a  new  joint  TD 
between  the  Netherlands  and  the  US. 


Planning/Design 

Execution 

Analysis 

Communication 

Coordination 

Collaboration 

Figure  1-2:  Collaboration  Matrix  for  Each  TD. 

The  follow-on  meetings  occurred  approximately  every  6  months,  over  a  three  year  period.  Meetings  centered 
around  one  or  more  tech  demos  associated  with  the  host  country.  Many  TDs  were  actual  live  tests  using  real 
assets  in  air  or  on  ground  or  on  water,  providing  needed  realism  and  hands  on  experience.  The  tech  demo 
researchers  presented  the  TD(s),  invited  specialists  as  desired,  and  used  the  available  time  to  discuss  and 
critique  the  demo  specifics.  Contrasting  approaches/concepts  were  also  discussed. 

As  a  means  to  disseminate  the  results  and  lessons  learned  from  this  Task  Group,  a  NATO  “Technology  Forum” 
Workshop  (RWS-217)  is  organized  at  the  end  of  this  effort.  This  forum  presents  summaries  of  all  TDs 
conducted  throughout  the  TG  period  through  posters,  videos,  and  hardware  demos/simulations.  Discussions 
center  around  lessons  learned  and  the  way  forward  regarding  multi-vehicle  control  by  a  single  operator. 


1.3  SUMMARY 

A  total  of  15  TDs  were  included  as  part  of  HFM-170.  These  TDs  are  listed  in  Figure  1-3,  along  with  the  Host 
country.  TDs  focused  on  many  critical  issues  including  multi-vehicle  control,  manned-unmanned  teaming, 
human-automation  interaction,  telepresence  interfaces,  delegation  interfaces,  vehicle  hand-offs,  operator 
workload  adaptive  systems,  variable  levels  of  autonomy,  authority  sharing,  situation  awareness  aids,  cognitive 
workload  assessment,  swarming  interface  technology,  and  dynamic  mission  management. 
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Technology 

Demonstration 

Title 

Host  Country 

1 

Multi-crew  Control  of  a  Single  Unmanned  Aircraft 

Canada 

2 

Behaviour  based  Collision  Avoidance  and  Formation 
Control  of  Multiple  Unmanned  Vehicles 

Canada 

3 

Supervisory  Control:  OmniSense 

Canada 

4 

Interacting  with  Multi-agent  Systems  /  UAV  Swarms 

France 

5 

PEA  Human  Factors  and  Authority  Sharing 

France 

6 

Cognitive  and  Cooperative  Automation  for  Aerial 
Manned-Unmanned  Teaming  Missions 

Germany 

7 

Remote  Auditory  Target  Detection  Using  an  Unmanned 
vehicle  -  Comparison  Between  a  Telepresence 
Headtracking  3D  Audio  Setup  and  a  Joystick-Controlled 
System  with  a  Directional  Microphone 

Netherlands 

8 

Supervisory  Control:  Optimal  Distribution  of  Workload 
Among  Operators  for  Mixed  Initiative  Control  of 

Multiple  UAVs 

Portugal 

9 

Task  Switching  for  Multi-UGV  Control 

Sweden 

10 

Supervisor  Control  of  UGVs  for  Tactical 

Reconnaissance 

Sweden 

11 

Dynamic  Airborne  Mission  Management  Capability 
Concept  Demonstrator 

United  Kingdom 

12 

Multi -UAV  Supervisory  Control  Interface  Technology 
(MUSCIT)  Demonstration 

United  States 

13 

Delegation  Control  of  Multiple  Unmanned  Systems 
(DELCON) 

United  States 

14 

Intelligent  Agents  as  Supervisory  Assets  for  Multiple 
Uninhabited  Systems:  RoboLeader 

United  States 

15 

Unmanned  Surface  Vehicle  Control  &  Monitoring 
Human-Computer  Interface  for  Amphibious  Operations 

United  States 

Figure  1-3:  HFM-170  Technology  Demonstrations. 

The  following  chapters  begin  with  an  extensive  review  of  the  efforts  undertaken  by  HFM-170  to  identify 
supervisory  control  frameworks  by  which  to  describe  the  research  being  done  in  this  area,  including  but 
not  limited  to  the  TDs.  This  is  followed  by  a  summary  of  each  TD  including  its  goals,  approach,  and  results/ 
lessons  learned. 


RTO-TR-HFM-170 


1  -5 


INTRODUCTION 


ORGANIZATION 


1.4  REFERENCES 

[1]  Uninhabited  Military  Vehicles  (UMVs):  Human  Factors  Issues  in  Augmenting  the  Force.  2007.  NATO- 
RTO-TR-HFM-07 8 .  Neully-sur-Seine:  NATO-RTO. 


1  -6 


RTO-TR-HFM-1 70 


ORGANIZATION 


Chapter  2  -  FRAMEWORKS  FOR  SUPERVISORY  CONTROL 
OF  MULTIPLE  UNINHABITED  MILITARY  VEHICLES 

Dr.  Christopher  A.  Miller 

Smart  Information  Flow  Technologies 
211  First  St.  N.  Suite  300 
Minneapolis,  MN  55401 
USA 

Email:  cmiller@sift.net 


2.1  OVERVIEW  AND  SCOPE 

Throughout  the  meetings  of  HFM-170,  we  have  discussed  the  possibility  and  desirability  of  having  a 
framework  for  describing  and  contrasting  the  various  supervisory  control  systems  that  each  member  group  has 
been  working  with,  as  well  as  others  we  have  experienced.  We  have  discussed  many  different  potential 
frameworks,  without  coming  to  consensus  about  any  specific  one  of  them.  The  purpose  of  this  document  is  to 
review  those  discussions,  along  with  conclusions  reached  about  the  nature  of  supervisory  control  frameworks 
and  desirable  attributes  for  this  application  (as  distinct  from  others),  as  well  as  to  review  the  different 
frameworks  proposed  and  their  strengths  and  weaknesses. 


2.2  FRAMEWORKS  AND  SUPERVISORY  CONTROL 

Almost  concurrently  with  Sheridan’s  defining  the  term  and  concept  “supervisory  control”,  he  proposed  a 
framework  for  characterizing  it  -  his  ten  stage  model  which  will  be  reviewed  in  Section  2.3  below. 
In  Sheridan’s  definition,  which  has  remained  more  or  less  intact  since  he  coined  it,  supervisory  control  is  any 
human-machine  relationship  in  which  the  machine  is  in  a  subordinate  state  like  that  of  a  human  supervisor- 
subordinate  relationship.  “...  [Supervisory  control  derives  from  the  close  analogy  between  a  supervisor’s 
interaction  with  subordinate  people  in  a  human  organization  and  a  person’s  interaction  with  intelligent 
automated  sub-systems.”  [1]. 

But  just  as  there  is  a  huge  range  of  human-human  supervisory  relationships  (from  master-slave  to  parent-child 
to  collaborative  work  team  to  president-nation  to  gang  leaders),  so  there  are  a  wide  range  of  human-machine 
supervisory  control  relationships  and  it  would  be  desirable  to  be  able  to  characterize  and  discuss  their 
similarities  and  differences.  If  for  no  other  reason,  it  would  be  helpful  to  be  able  to  discriminate  one  from 
another,  especially  when  attempting  to  determine  which  type  works  best  for  which  application.  So,  beginning 
well  before  Sheridan,  there  has  been  a  long  history  of  characterizing  the  types  or  stages  of  a  human-machine 
supervisory  control  relationship. 

But  it  is  worth  remembering,  after  George  Box  [2],  that  ‘all  frameworks  are  wrong,  but  some  frameworks  are 
useful’.  Box  uttered  that  quote,  he  was  talking  about  models,  but  a  framework  is  essentially  a  model  -  and 
perhaps  a  taxonomy.  Box  meant  that  a  model  will  never  fully  capture  the  details  and  intricacies  of  the  real 
world,  but  that  some  models  may  capture  interesting  or  relevant  aspects  of  it  -  and,  in  fact,  by  eliminating 
excess  detail,  some  models  may  even  make  it  easier  to  see  relevant  relationships  and  distinctions.  Similarly, 
there  are  always  multiple  ways  of  parsing  any  complex  phenomenon  into  multiple  frameworks  or  taxonomies. 
None  of  these  will  capture  the  full  richness  of  the  phenomenon,  but  some  of  them  -  indeed,  multiple  versions 
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-  may  be  useful  because  they  organize  the  phenomenon  in  helpful,  insightful  ways,  highlighting  some 
similarities  and  differences  while  obscuring  others.  There  is  no  single  “right”  framework  for  describing 
supervisory  control,  but  different  ones  may  be  more  or  less  useful  for  different  purposes.  Therefore,  for  any 
framework  development  or  evaluation,  it  helps  to  begin  with  a  discussion  of  the  purposes  or  goals  which  the 
framework  will  serve. 

2.2.1  Why  Have  a  Framework? 

There  are  a  wide  set  of  possible  reasons  one  might  want  to  have  a  framework  for  supervisory  control.  These 
include: 


•  Training  -  A  framework  can  serve  as  a  “mall  map”  for  training  users  in  the  specific  attributes  of  a 
single  system,  or  in  training  the  similarities  and  differences  across  multiple  systems. 

•  Interaction  -  The  “mall  map”  attributes  of  a  framework  can  also  serve  as  a  mental  model  to  aid  users 
in  understanding  and  remembering  the  attributes  and  behaviors  a  supervisory  control  system  affords. 

•  Organizing  Investigation  -  For  research  purposes,  a  framework  can  serve  as  a  map  to  “uncharted 
territory”,  helping  to  determine  what  areas  have  been  investigated  and  what  have  not  -  and  can  aid  in 
generalizing  results  if  it  highlights  the  similarities  and  differences  between  “regions”  of  the  space  of 
possible  system  alternatives. 

•  Understanding  Alternatives  for  Design  -  As  investigation  of  the  space  of  alternative  approaches  to 
supervisory  control  are  completed,  they  form  a  set  of  data  about  design  alternatives  -  a  database  that 
may  be  organized  according  to  a  framework.  Thus,  the  framework  may  serve  as  a  guide  to  designers 
to  understand  both  what  alternative  design  methods  are  available  and  what  conditions  they  have  been 
proven  to  work  well  in. 

The  reasons  for  this  working  group  were  related  to  the  later  two  -  to  characterize  the  set  of  systems  and 
applications  we  were  discussing  as  a  part  of  our  collaboration.  To  the  degree  that  the  framework  we  created 
helped  us  gain  insights  from  the  set  of  supervisory  control  applications  we  studied,  that  would  be  added 
benefit.  Finally,  we  also  wanted  an  organizing  principle  for  this  report. 

2.2.2  Framework  Attributes  and  Goals 

Frameworks  can  be  more  or  less  elaborate,  and  they  can  highlight  or  suppress  different  aspects  of  the 
phenomenon  they  model.  Given  the  goals  described  above,  desirable  attributes  for  our  framework  included: 

•  Brevity  and  Simplicity  -  To  serve  as  an  organizing  principle  for  communicating  the  results  of  this 
working  group  to  the  outside  world,  it  was  important  that  the  framework  be  simple  and  brief  enough  to 
be  conveyed  and  explained  to  a  reading  audience  in  a  limited  amount  of  time.  We  were  willing  to  make 
some  sacrifices  in  the  coverage  or  resolution  of  the  model  in  order  for  it  to  be  readily  comprehensible  by 
our  audience.  To  some  degree,  the  “fame”  (or  prior  knowledge)  of  a  candidate  framework  could  be  used 
to  compensate  for  simplicity,  and  some  frameworks  (most  notably  Sheridan’s  [3],  [5]  and  [6])  were 
already  well  known  in  the  human  factors  and  engineering  world. 

•  Emphasis  on  Domain  -  Since  this  exercise  was  a  part  of  a  NATO  RTO  working  group  on  supervisory 
control  of  unmanned  military  vehicles,  this  colored  our  efforts  in  three  ways: 

•  First,  “supervisory  control”  was  the  focus  and  topic  of  our  framework  development  efforts.  While 
many  other  topics  in  the  domain  of  human-automation  interaction  are  related  to  supervisory 
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control  (such  as  levels  of  autonomy,  adaptive  autonomy  and  user  interface  and  interaction 
design),  these  were  not  directly  our  focus. 

•  Second,  our  emphasis  was  on  developing  a  framework  for  characterizing  supervisory  control  of 
Unmanned  Military  Vehicles  (UMVs).  While,  as  will  be  seen  below,  we  occasionally  discussed 
other  forms  of  supervisory  control  (both  human-human  and  other  application  domains), 
our  ultimate  emphasis  was  on  UMVs. 

•  Third,  since  supervisory  control  is,  at  root,  a  relationship  between  human  and  automation,  we 
focused  on  characterizing  and  describing  alternate  ways  in  which  that  relationship  could  exist. 
Technology  and  its  application  domains,  while  relevant,  were  not  the  primary  focus. 

•  Coverage  -  It  was  important  that  the  framework  developed  be  able  to  cover  -  that  is,  to  represent  and 
describe  -  each  of  the  applications  that  were  being  developed  and  tested  by  the  various  participating 
members  and  their  countries’  laboratories.  Coverage  outside  that  set  was  nice  to  have,  but  deemed 
less  important. 

•  Distinction  -  While  the  ability  to  cover  the  set  of  demonstrations  being  developed  by  the  working 
group  participants  was  important,  it  was  equally  important  that  our  framework  be  able  to  make 
distinctions  between  them.  We  wanted  to  be  able  to  organize  and  characterize  these  demonstration 
systems,  identifying  their  similarities  and  differences. 


2.3  ALTERNATE  FRAMEWORKS  CONSIDERED 

In  this  section,  we  will  describe  the  various  frameworks  considered  during  the  course  of  the  working  group. 

Sheridan 

Sheridan’s  initial  “framework”  for  characterizing  supervisory  control  relations  was  a  spectrum 
arrayed  as  a  10  item  list  whose  endpoints  are  full  control  autonomy  for  the  human  (essentially  no  role 
for  automation)  and  vice  versa  [3].  The  intermediate  levels  in  this  spectrum,  then,  represent  alternative 
forms  of  supervisory  control  interactions.  A  version  of  this  list  is  shown  in  Table  2-1. 


Table  2-1 :  Levels  of  Automation  (after  [3]). 


1 

Human  does  it  all. 

2 

Computer  offers  alternatives 

3 

Computer  narrows  alternatives  down  to  a  few 

4 

Computer  suggests  a  recommended  alternative 

5 

Computer  executes  alternative  if  human  approves 

6 

Computer  executes  alternative;  human  can  veto 

7 

Computer  executes  alternative  and  informs  human 

8 

Computer  executes  selected  alternative  and  informs  human  only  if  asked 

9 

Computer  executes  selected  alternative  and  informs  human  only  if  it  decides  to 

10 

Computer  acts  entirely  autonomously. 

RTO-TR-HFM-170 


2-3 


FRAMEWORKS  FOR  SUPERVISORY  CONTROL 
OF  MULTIPLE  UNINHABITED  MILITARY  VEHICLES 


Strengths  and  Weaknesses 

Sheridan’s  spectrum  is  very  simple  and  very  well  known,  but  it  has  several  flaws  as  a  framework  for  our  needs. 
It  has  been  criticized  (in  Parasuraman,  Sheridan  and  Wickens  [6],  among  others)  for  confounding  automation 
employed  in  the  presentation  of  information,  in  the  making  of  decisions  and  in  the  executing  of  actions.  Because 
most  military  systems  are  complex  enough  to  employ  automation  operating  on  many  different  tasks  and  task 
types  concurrently,  when  one  assigns  a  number  using  Sheridan’s  framework,  one  is  forced  to  either  be 
ambiguous  about  the  task  or  operational  domain  being  described  by  the  number,  to  break  tasks  down  to  a  fine 
enough  level  where  only  one  of  Sheridan’s  levels  makes  sense  as  a  descriptor,  or  to  “average”  over  the 
automation  levels  and  the  tasks  and  applications  where  automation  is  provided.  This  argument  is  made  in  more 
detail  in  Miller  and  Parasuraman  [4], [9]. 

Parasuraman,  Sheridan  and  Wickens 

Sheridan’s  spectrum  model  is  essentially  uni-dimensional.  As  noted  above,  though,  it  achieves 
“unidimensionality”  by  combining  several  potential  behaviors  that  automation  can  perform.  While 
extremely  simple  and  intuitive,  this  unidimensionality  may  well  be  too  simple  to  make  the  kinds  of 
distinctions  between  systems  and  applications  which  we  would  like  to  make. 

Another  problem  with  uni-dimensional  models  of  human-automation  relationships  is  that  they  are  ambiguous 
about  what  the  application  domain  of  the  relationship  applies  to.  Parasuraman  et  al.  [6]  noted  that  Sheridan’s 
levels  referred  mainly  to  automation  that  makes  decisions,  offers  suggestions  and/or  executes  actions.  There 
are,  however,  other  jobs  automation  can  do:  for  example,  sensing  and  analyzing  information  to  detect  situations 
of  interest,  without  necessarily  offering  any  advice  on  what  to  do  with  the  information.  Parasuraman  et  al.  [6] 
applied  a  simple,  stage  model  of  human  information  processing  to  arrive  at  four  functions  that  must  be 
accomplished  to  perform  most  tasks: 

1)  Information  acquisition; 

2)  Information  analysis; 

3)  Decision  and  action  selection;  and 

4)  Action  implementation. 

Since  these  functions  can  be  performed  by  either  human  or  automation  in  various  mixes,  in  effect 
Parasuraman  et  al.,  [6]  added  a  second  dimension  to  Sheridan’s  spectrum  -  that  of  the  function  or  task  the 
relationship  is  defined  over.  Most  human  +  automation  systems  can  be  characterized  by  a  mix  of  LOAs  across 
these  four  functions,  as  in  Figure  2-1.  One  system  (A)  might  be  highly  autonomous  in  information  acquisition, 
but  comparatively  low  on  the  other  functions,  while  a  second  (B)  might  offer  a  high  LOA  across  all  four 
functions. 
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Figure  2-1 :  Levels  of  Automation  by  Information  Processing  Phase  for  Two  Systems  (from  [5]). 


Strengths  and  Weaknesses 

This  information  processing  stage  X  autonomy  level  model  has  gained  extensive  acceptance  in  the  research 
community  and  systems  have  begun  to  be  developed  in  accordance  with  it.  Research  has  shown  different 
effects  of  automation  at  the  different  stages  [7],  Nevertheless,  it  was  felt  that  it  too  was  too  coarse  grained  to 
provide  adequate  distinctions  between  the  various  supervisory  control  systems  being  explored  by  members  of 
HFM-170.  Furthermore,  some  members  of  the  group  had  reported  prior  attempts  to  use  the  Parasuraman, 
Sheridan  and  Wickens  framework  to  describe  and  define  systems  and  had  encountered  difficulties  in  making 
clean  distinctions  between  the  information  processing  stages. 

2.3.1  Miller  and  Parasuraman 

An  implication  of  the  Parasuraman  et  al.  [6]  levels  x  functions  model  is  that  a  parent  task1  can  be  decomposed 
and  that  a  single  automation  level  need  not  be  applied  homogenously  across  the  sub-tasks.  However,  in  their 
model  a  parent  task  is  decomposed  into  abstract  sub-functions  based  on  information  processing  stages, 
whereas  other  decomposition  methods  might  arguably  provide  more  insight  into  how  a  task  may  be 
performed.  In  fact,  the  role  of  task  analysis  in  Human  Factors  is  to  perform  exactly  such  decompositions  in  a 


1  Note  that  in  Sheridan’s  model,  as  well  as  in  Parasurman,  Sheridan  and  Wickens,  and  all  of  the  other  models  discussed  in  this 
document,  a  decomposition  of  the  functions  to  be  performed  by  human(s)  and  automation  is  important,  but  it  is  less  important 
whether  the  focus  of  analysis  is  prescriptive  tasks,  abstract  functions  or  even  the  goals  which  are  accomplished  by  those  tasks  and 
functions.  Since  supervisory  control  necessarily  concerns  itself  with  allocating  the  work  of  multiple  agents  for  the  accomplishment 
of  a  goal,  allocation  will  necessarily  include  goals,  functions  and  the  tasks  or  methods  which  accomplish  them.  Most  of  the  models 
described  in  this  document  apply  regardless  of  whether  the  supervisor  allocates  via  prescriptive  tasks  or  ecological  abstract 
functions  and  goals. 
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hierarchical  fashion  through  any  number  of  levels  to  some  primitive,  “stopping”  level  [8]  that  may  be  imposed 
by  biology,  physics  or,  more  commonly,  the  purpose  of  the  decomposition. 

In  a  2003  paper,  Miller  and  Parasurman  [9]  argued  that  while  the  two-dimensional  LOA  model  offered  by 
Parasuraman  et  al.  [6]  represents  a  major  advance  over  earlier  uni-dimensional  models,  it  arguably  does  not  go 
far  enough.  The  subdivision  of  a  parent  task  into  four  information  processing  phases  represents  only  a  single 
level  of  decomposition  into  abstract  task  categories.  In  practice,  tasks  are  accomplished  by  hierarchically 
decomposable  sequences  of  specific  activities  -  the  parent  task’s  sub-tasks.  Automation  may  be  applied 
differently  to  each  and  every  sub-task  that  comprises  the  parent  task.  Thus,  the  profile  of  automation  levels 
sketched  in  Figure  2-2  could  stretch  instead  over  as  many  sub-tasks  and  levels  as  we  want  or  need  to  divide  a 
parent  task  into. 


Figure  2-2:  Hypothetical  Decomposition  of  Task  A  Into  a  Hierarchical  Set 
of  Sub-Tasks  -  Each  of  Which  May  Have  Differing  Automation  Levels. 


In  fact,  the  relationship  between  automation  “level”  and  task  decomposition  is  more  complex  still.  As  is  well 
understood  [6],  automation  does  not  merely  shift  responsibility  for  tasks  but  can  change  their  nature  as  well. 
In  a  task  decomposition,  this  means  that  some  sub-tasks  may  be  eliminated  while  others  are  added. 
This  implies  that  there  will  generally  be  multiple  alternate  decompositions  depending  on,  among  other  things, 
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what  LOA  is  used.  Each  alternative  constitutes  a  different  combination  of  human  and  automation  sub-tasks 
and,  thus,  a  different  method  of  accomplishing  the  parent  task. 

When  one  identifies  a  LOA  for  a  complex  system  using  Sheridan’s  dimensions,  one  is  in  principle  identifying 
something  like  an  average  or  modal  level  over  the  sub-tasks  the  human  +  automation  system  accomplishes. 
Similarly,  when  one  uses  a  levels  X  stages  model  such  as  in  Parasuraman  et  al.  [6],  one  is  performing  an 
abstract  and  coarse-grained  decomposition  of  the  parent  task  into  sub-tasks  clustered  by  information 
processing  stage.  Assigning  levels  by  sub-task  stages  offers  more  sensitivity  than  assigning  them  only  to  the 
parent  task,  but  it  is  still  an  abstraction.  In  practice,  one  could  identify  the  specific  sub-tasks  to  be  performed 
and  represent  an  automation  level  for  each  of  them. 

Figure  2-2  provides  a  hypothetical  illustration  of  this  relationship.  A  parent  task  “A”  might  be  said  to  have 
a  certain  level  of  automation  on  Sheridan’s  scale,  but  in  fact,  that  task  is  comprised  of  a  series  of  sub-tasks 
(Al  -  A6)  each  of  which  may  have  a  different  mix  of  human  and  automation  involvement  (as  illustrated  by 
the  different  shadings  of  the  sub-task  boxes).  These  sub-tasks  can  be  reasonably  organized  or  clustered  into 
Parasuraman,  Sheridan  and  Wickens’s  information  processing  stages,  but  this  can  obscure  both  the  fact  that 
specific  sub-tasks  exist  and  that  they  may  have  different  mixtures  of  human  and  automation  involvement. 
Furthermore,  each  of  the  tasks  at  this  level  can  also,  generally,  be  further  decomposed  into  sub-sub-tasks 
(Al.l  -  A6.2)  which  may  also  have  different  mixes  of  human  and  automation  involvement...  and  so  on,  until 
some  desired  primitive  level  is  reached. 

Why  would  one  want  to  perform  this  kind  and  level  of  analysis?  More  and  finer-grained  sub-tasks  are  not 
necessarily  better  and,  in  fact,  Parasuraman  et  al.  [6]  explicitly  state  that  they  chose  a  four-stage  model  to 
simplify  design  considerations.  Precision  may  be  inherently  desirable  for  some  purposes  (such  as  training  and 
detailed  design),  but  Miller  and  Parasuraman’s  [9], [4]  purpose  in  achieving  greater  precision  and  finer 
granularity  was  to  support  flexible  task  delegation.  As  we  saw  above,  for  any  intermediate  LOA  for  a  task, 
there  are  roles  for  both  humans  and  automation  in  its  sub-tasks.  Yet,  someone  must  coordinate  those  roles. 
Insofar  as  human  supervisors  are  required  to  manage,  or  at  least  be  aware  of,  that  division  of  labor,  they  must 
understand  the  decomposition  of  the  task  and  of  the  allocation  and  coordination  requirements  among  its  sub¬ 
tasks.  Supervisory  control  is  a  process  of  task  delegation  and  delegation  requires  task  decomposition. 

Strengths  and  Weaknesses 

That  said,  while  precision  and  flexibility  in  stipulating  how  a  task  is  to  be  performed  is  a  necessary  aspect  of 
powerful  delegation  relationships,  the  goals  of  a  framework  for  this  working  group  were  somewhat  different. 
Instead  of  precisely  defining  each  and  every  difference  in  how  various  systems  achieve  supervisory  control,  we 
wanted  to  group  similar  systems  -  a  process  that  implies  some  degree  of  ignoring  (or,  at  least,  clustering) 
differences.  Furthermore,  prior  work  with  the  Miller  and  Parasuraman  [9]  approach  (as  well  as  the  similar  LoA3 
and  CLAMP3  frameworks  described  below)  has  shown  that,  while  powerful,  they  are  as  cumbersome  and  time 
consuming  to  use  for  the  purposes  of  description  as  most  task  analytic  techniques  in  human  factors  [8], [10]. 
Thus,  they  were  inappropriate  for  the  purposes  of  briefly  and  intuitively  summarizing  the  similarities  and 
differences  between  supervisory  control  systems  being  explored  in  the  HFM-170  technology  development 
efforts. 


2.4  LoA3 

During  prior  work  for  the  U.S.  Air  Force  Research  Laboratory  and  in  a  proposal  to  the  U.S.  Army’s 
Aeroflightdynamics  Directorate  (AFDD),  Miller  et  al.,  expanded  the  concept  of  hierarchical  task  decomposition 
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to  describe  supervisory  control  to  develop  what  they  called  the  “LoA3”  concept.  This  stood  for  levels  of 
authority,  abstraction  and  aggregation,  and  this  triumvirate  of  parameters  was  advanced  as  a  way  of  describing 
and  defining  delegation  relationships.  Delegation,  in  general,  was  defined  as  illustrated  in  Figure  2-3. 


Delegation  means  ...  Abstraction  Tasks 


Giving  to  a  subordinate 
the  responsibility  to 


perform  a  task 


Along  with  some 


authority  |not 
necessarilybomplete) 
over  how  to  perform 
that  task 

>  Instructions  can 
constrain  this  authorft 

#■  And  some  authority  to 


to  perform  the  task. 


Tasks  delegated  Supervisor’s  Tasks 
to  subordinate 

Authority  Level  and  Resources  may 
also  be  varied  by  the  supervisor 


Figure  2-3:  A  3-Dimensional  Definition  and  Framework  for  Delegation  Interactions. 


The  three  dimensions  were: 

1)  Abstraction  -  This  is  essentially  the  means-ends  or  hierarchical  decomposition  of  a  task.  Each  parent 
task  represents  the  “ends”  or  goal  of  a  child  (or  set  of  child)  tasks;  each  child  task  represents  the 
means  by  which  a  parent  task  is  accomplished.  Delegation  means  handing  over  some  responsibility 
for  some  tasks  which  themselves  are  part  of  larger,  parent  tasks  and  which  also  decompose  into 
smaller  child-  or  sub-tasks.  Delegating  the  task,  at  any  level,  means  transferring  some  authority  to 
perform  that  task  (if  any  workload  is  to  be  saved)  and  authority  over  some  resources  (at  least 
attentional  resources)  to  perform  the  task  and  its  sub-tasks  and  to  make  the  decisions  about  which 
sub-task  methods  to  pursue. 

2)  Aggregation  -  This  is  essentially  the  part-whole  decomposition  of  resources.  Any  effective  act  of 
delegation  must  include  the  delegation  of  (at  least  partial)  control  over  resources  if  it  is  to  accomplish 
anything  -  even  if  the  resources  delegated  are  only  the  attentional  and  decision  making  resources  of 
the  subordinate.  Of  the  full  set  of  resources  which  a  supervisor  controls,  s/he  will  delegate  some 
control  over  some  of  those  resources  to  the  subordinate. 

3)  Authority  -  Authority  represents  the  degree  of  autonomy  that  the  subordinate  has  over  the  tasks 
(abstraction  dimensions)  and  resources  (aggregation  dimension)  he/she/it  has  been  delegated. 
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Sheridan’s  10  levels  (cf.  Figure  2-1  above)  can  be  thought  of  as  a  spectrum  of  autonomy  - 
characterizing  a  range  from  “computer  has  no  autonomy”  through  varying  levels  of  autonomy  to 
provide  suggestions  only,  autonomy  to  act  only  if  authorized  to  do  so  immediately  prior,  authorized  to 
act  only  if  the  action  is  reported,  and  ending  at  “computer  has  full  autonomy”. 

These  three  dimensions,  then,  let  us  specify  an  act  of  delegation  as  depicted  in  Figure  2-4.  Any  delegation 
involves  the  transfer  of  responsibility  for  some  sub-tasks  or  tasks  for  which  the  supervisor  has  responsibility, 
along  with  some  resources  over  which  the  supervisor  has  control.  In  both  cases,  the  transfer  of  authority  may 
not  be  complete,  even  over  these  sub-tasks.  The  supervisor  may  require  the  subordinate  to  coordinate, 
ask  permission,  inform,  etc. 


Tasks 


Resources 


7 


Re  sour c  e  s  del  eg  ate  d 
to  subordinate 


Supervisors 

resources 


Tasks  delegated 
to  subordinate 


Supervisors  Tasks 


Authority  is  associated  with  how  the  subordinate  is 
authorized  to  make  task  or  resource  decisions 


A  supervisor  delegates  responsibility  to  perform  some  task 
(with  some  level  of  authority  )  using  some  resources  (with 

some  level  of  authority) 


Figure  2-4:  An  Act  of  Delegation  Can  Be  Specified  as  a 
Transfer  of  Responsibility  Along  Three  Dimensions. 


Note  that  if  the  delegation  act  extends  over  time  and  functions,  it  establishes  a  supervisory  control  relationship 
between  a  supervisory  and  a  subordinate.  Therefore,  the  three  dimensions  of  the  LoA3  model  define  a  three 
dimensional  “space”  within  which  we  could  place,  in  principle,  any  supervisory  control  relationship  (or  act)  - 
as  illustrated  in  Figure  2-5. 


RTO-TR-HFM-170 


2-9 


FRAMEWORKS  FOR  SUPERVISORY  CONTROL 
OF  MULTIPLE  UNINHABITED  MILITARY  VEHICLES 


Autonomy/  Authority 


#  Level  of  Abstraction-  At  what  level(s)  can  the  human  command;  in 
terms  of  task  hierarchy-  with  what  level  of  authority? 

#  Level  of  Aggregation-  what  resources  (how  many  agents/systems) 
can  the  human  command- with  what  level  of  authority? 

#  Level  of  Autonomy  or  Authority-  How  independently  can  the 
automation  act?  Sheridan's  spectrum 

These  are  dimensions  that  can  be  varied  in  LoA  experiments 

Figure  2-5:  A  3-Dimensional  “Delegation  Space”  Formed  by  the  LoA3  Dimensions. 


Strengths  and  Weaknesses 

The  LoA3  model  provides  a  rich  description  of  the  act  of  delegation,  but  there  were  problems  with  it  for  the 
purposes  of  our  technical  group.  First  and  foremost,  it  relies  on  the  same  hierarchical  task  decomposition  as 
the  Miller  and  Parasuraman  approach  described  in  Section  2.3.1  above.  As  discussed  there,  this  approach 
essentially  requires  a  detailed  and  complete  task  decomposition  for  each  alternative  considered.  While  that 
may  be  useful  for  design  and  for  a  deep  understanding  of  system  alternatives,  it  is  not  compatible  with  the 
easily  understood  taxonomic  groupings  this  working  group  was  after.  Second,  this  model  was  seen  as  having  a 
failing  in  that  it  concentrated  exclusively  on  the  delegatory  act  and  had  nothing  to  say  about  the  environment 
or  technological  context  in  which  that  delegation  was  performed.  Since  the  members  of  HFM-170  were 
working  on  characteristically  different  technologies  and  domains  (e.g.,  ground  vs.  air  vs.  sea  UMVs),  it  was 
felt  that  an  adequate  framework  for  the  group  needed  to  reflect  these  differences  and  similarities  as  well. 

2.5  CLAMP3 

CLAMP3  was  an  attempt  to  remedy  the  second  failing  of  the  LoA3  model  as  described  above  -  to  embed  the 
LoA3  model  in  a  broader  framework  which  would  include  the  ability  to  characterize  aspects  of  the  technology 
and  application  domains  of  the  supervisory  control  systems  it  classified.  CLAMP3  was  developed  by  Harry 
Funk  and  used  as  the  framework  for  a  simulation  and  testing  environment  for  delegation  systems  which  was 
built  and  used  initially  by  Jay  Shively,  Susan  Flaherty  and  Lisa  Fern  at  the  U.S.  Army’s  Aeroflightdynamics 
Directorate  (AFDD)  and  later  expanded  and  used  for  additional  experiments  by  personnel  at  Smart 
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Information  Flow  Technologies  (a  small  U.S.  business)  and  George  Mason  University  under  funding  from 
AFDD. 

CLAMP3  stands  for  C3  (three  dimensions  of  Context)  +  LoA3  +  Mapping  for  Predicting  of  Personnel  Performance 
-  as  illustrated  in  Figure  2-6.  In  other  words,  CLAMP3  takes  the  3  levels  describing  the  delegation  interaction 
from  LoA3  and  “wraps”  them  in  a  description  of  the  context  in  which  that  delegation  action  takes  place  (the  three 
context  dimensions)  along  with  a  description  of  the  resulting  performance  metrics  for  the  human-machine 
system. 


(Complexity, 

Capabilities^, 

Capabilities^) 


Authority, 

Abstraction, 

Aggregation) 


Figure  2-6:  The  CLAMP3  Model  -  Embedding  LoA3  in  Context  and  Performance  Outcomes. 


The  three  context  dimensions  used  were: 

1)  A  description  of  the  Situation  Complexity  -  That  is,  for  example,  is  the  UAV  being  asked  to  fly  straight 
and  level  at  cruising  altitude  on  a  clear  and  windless  day,  or  is  it  being  asked  to  fly  nap  of  the  earth  at 
night,  in  storms,  with  enemy  radars  and  small  arms  fire. 

2)  A  description  of  the  Capabilities  of  the  Operator  -  Is  s/he  a  trained  fighter  pilot  (training  that  might  be 
helpful  for  operating  a  UAV,  but  useless  or  even  counter-productive  for  operating  a  UGV)  or  an 
untrained  infantryman?  Is  s/he  operating  in  a  quiet  room  devoting  full  attention  to  the  UMV 
management  task,  or  is  s/he  engaged  in  combat,  taking  fire  and  perhaps  riding  in  the  back  of  an  armored 
vehicle  in  rough  terrain? 

3)  A  description  of  the  Capabilities  of  the  Unmanned  Vehicle  -  These  could  be  any  relevant  attribute  of 
the  UMV,  but  particular  relevance  will  generally  be  associated  with  control  and  functional 
capabilities.  A  vehicle  which  only  ever  does  one  thing  (e.g.,  flies  in  a  circle  transmitting  images)  will 
impose  much  less  burden  on  an  operator  than  one  that  admits  many  different  behaviors  and  modes. 
Similarly,  one  whose  performance  and  stability  is  unreliable  will  require  much  more  human  attention 
than  one  that  is  highly  reliable. 

At  the  “other  end”  of  CLAMP3  is  a  description  of  the  outcome  or  effects  of  a  delegation  relationship  within 
the  context  described  -  that  is,  performance  measures  in  terms  of  both  mission  and  human  performance. 
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Strengths  and  Weaknesses 

While  CLAMP3  has  been  used  as  the  framework  for  experimental  work  sponsored  by  AFDD,  it  retains 
problems  for  use  as  a  framework  for  this  working  group.  First,  it  inherits  the  problem  of  LoA3  and  of  Miller 
and  Parasuraman  [9]  in  that  it  is  based  on  a  hierarchical  task  decomposition  to  describe  the  delegation 
relationship  between  human  and  automation.  While  this  is  rich  and  detailed,  it  is  likely  too  rich  and  detailed 
for  convenient  use  by  this  group  or  easy  understanding  by  others. 

More  importantly,  while  CLAMP3  calls  attention  to  the  need  to  situate  a  description  of  a  supervisory  control 
relationship  in  a  context  and  to  describe  its  effectiveness,  methods  of  representing  these  dimensions  are 
underspecified.  CLAMP3  tells  us,  for  example,  that  it’s  important  to  consider  the  complexity  of  the  context, 
but  gives  us  no  metric  or  even  set  of  factors  that  might  contribute  to  that  complexity.  Such  would  be  necessary 
for  comparing  applications  within  and  across  the  members  of  HFM-170.  The  main  contribution  of  CLAMP3 
was  to  remind  us  to  include  these  dimensions  as  we  moved  forward  in  trying  to  specify  a  framework  for  use 
by  this  group. 


2.6  7D 

There  was  general  consensus  that  the  core  ideas  of  the  CLAMP3  framework  were  moving  in  the  right  direction 
-  that  is,  particularly  the  idea  that  folding  a  description  of  a  supervisory  control  relationship  into  a  description 
of  the  context  in  which  the  relationship  was  used  to  describe  the  resulting  system.  What  was  needed,  it  was 
felt,  was  a  way  of  reducing  the  complexity  of  the  associated  dimensions  while  regularizing  and  scaling  them. 
It  was  with  these  goals  in  mind  that  Chris  Miller  proposed  a  seven  dimensional  model  at  the  Stockholm 
meeting  of  HFM-170  in  June,  2008. 

The  seven  dimensions  of  this  model  were  formed  by  returning  to  the  LoA3  and  C3  (context)  dimensions  of  the 
CLAMP3  framework  and  attempting  to  characterize  and  scale  them  to  develop  a  multi-dimensional 
description  of  a  supervisory  control  relationship  plus  the  environment  in  which  that  relationship  occurred. 
More  specifically,  the  goal  was  to  use  the  previously-identified  abstract  dimensions  to  form  a  quantified  and 
specific  set  of  relevant  and  important  dimensions  along  which  the  HFM-170  projects  and  applications  varied. 

The  LoA3  scales  (autonomy,  abstraction  and  aggregation)  collectively  characterize  the  interaction  between  the 
human  and  automation,  as  we  noted  above.  The  C3  scales  (complexity,  operator  capabilities,  automation/UMV 
capabilities)  collectively  characterize  the  environment  in  which  the  LoA  relationship  exists  and  is  exercised. 
Using  this  insight,  Dr.  Miller  went  through  each  of  the  scales  and  attempted  to  distill  what  was  “important  or 
significant”  about  each  of  them  with  regard  to  the  supervisory  control  applications  being  developed  and 
discussed  by  this  working  group.  This  gave  us  a  means  for  identifying  dimensions  to  include  in  our  resulting 
model  and,  more  importantly,  for  creating  a  scale  for  each  dimension.  The  set  of  dimensions  identified  are 
depicted  in  Figure  2-7  (which  also  illustrates  how  they  were  derived  from  the  LoA3  and  C3  dimensions)  and 
are  discussed  along  with  the  scales  developed  to  represent  them  below. 
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LoAs 

Abstraction 

Aggregation 

Autonomy 


Proposed  Dimensions 

Task  Duration 
Task  Diversity 
#  Vehicles/ 

Subsystems 
Autonomy  — 


Labels 

(T)  Time  Span 
(D)  Diversity 

(VS)  Vehicles/ 
Subsystems 

(A)  Autonomy 


Cs 

Complexity  — 
Op  Capabilities 
UV  Capabilities 


Behavior  Change 

Frequency  (BFrq)  Behavior 

Operator  Ratio  .  frecl 

Intervention  Frequenqf°P)  °Perator 

Ratio 

(IFrq)  Intervent. 
Frequency 


Figure  2-7:  The  Seven  Descriptive  Dimensions  and  Their  Derivation 
from  the  LoA3  and  C3  Dimensions  of  the  CLAMP3  Model. 


In  general,  the  scales  were  created  to  capture  the  range  of  variation,  yet  show  interesting  degrees  of  difference, 
along  the  dimensions  we  saw  in  supervisory  control  UMV  systems  under  consideration  by  the  group. 
A  secondary  motivation  was  an  attempt  to  synchronize  the  length  and  scalar  values  on  each  of  the  scales  to 
facilitate  later  visualizations.  To  achieve  this  later  goal,  we  developed  seven  point  scales  for  each  dimension 
(as  described  below)  and  arrayed  each  of  the  scales  from  “worse”  (notionally  less  competent  systems)  to 
“better”  (notionally  more  competent  systems).  The  specific  divisions  are,  of  course,  somewhat  arbitrary  and 
debatable,  but  the  intention  was  to  provide  a  “chunking”  of  the  dimension  into  seven  significantly  different 
categories: 

•  What’s  important/significant  about  “Abstraction”?  Abstraction,  in  a  task  hierarchy  sense,  captures  the 
number,  types  and  relations  of  tasks/behaviors  the  UMV  is  designed  for.  If  the  top  level  task  in  a 
hierarchy  for  a  given  UMV  can  be  thought  of  as  “Perform  Mission”,  then  a  complete  decomposition 
will  represent  all  the  possible  tasks  that  system  will  perform.  The  “size”  of  that  hierarchy  tells  us 
important  things  about  the  mission(s)  and  capabilities  of  the  UMV  and  led  to  proposing  two  different 
dimensions: 

•  Mission/Task  Duration  (T)  -  Duration  of  missions  is  a  reasonable  stand-in  for  “size”  of  the 
hierarchy  -  how  much  time  span  does  a  typical  mission  or  task  being  analyzed  cover?  This  is  not 
the  task  that  the  operator  delegates,  but  rather  the  operational  window  for  the  UMV  itself. 
Length/duration  is  a  simple  dimension  ranging  from  seconds  to  days  or  weeks.  A  scale  of 
interestingly  different  levels  on  this  dimension  (for  the  set  of  UMV  systems  we  were  considering) 
was  proposed  as: 
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1)  Seconds 

2)  1-5  minutes 

3)  5-30  minutes 

4)  30-90  minutes 

5)  1.5 -6  hours 

6)  6-24  hours 

7)  Days 

•  Task  Diversity  (D)  -  Task  diversity  is  a  necessary  second  dimension  to  identify  the  “complexity” 
of  missions  and  of  UMV  roles.  How  many  different  types  of  tasks  is  the  UMV  involved  in? 
How  many  conceptually  distinct  functions2  are  performable  by  the  vehicle(s)?: 

1)  1  only 

2)  2-3 

3)  4-6 

4)  7-10 

5)  11-15 

6)  16-25 

7)  25+ 

•  What’s  important  about  “Aggregation”?  Aggregation  refers  to  the  number  of  vehicles  (or  vehicle 
“parts”  or  sub-functions)  being  controlled  by  the  user(s)  in  an  application  to  be  characterized  by 
the  framework.  Some  supervisory  control  systems  are  being  designed  to  control  multiple  UMVs 
simultaneously,  while  others  are  controlling  at  most  a  single  sub-system.  This  gave  rise  to  the  Vehicles/ 
Sub-systems  (VS)  dimension. 

•  Vehicles/Sub-Systems  (VS)  -  The  VS  dimension  captures  how  many  UMVs  and/or  UMV  sub¬ 
systems  are  typically  involved  in  a  mission  (in  the  analyst’s  focus  of  interest): 

1)  Single  sub-system 

2)  Multiple  (2  -  4)  sub-systems,  but  not  whole  vehicle 

3)  One  whole  vehicle  or  4+  sub-systems 

4)  2  whole  vehicles  (or  parts  thereof) 

5)  3-4  vehicles 

6)  4 -12  vehicles 

7)  Swarms  (12+) 

•  What’s  important  about  “Autonomy”?  Who’s  in  charge,  who  is  leading/following?  For  the  mission  as 
a  whole,  what’s  the  relationship  between  human  and  automation? 

•  Autonomy  (A)  -  To  characterize  this  dimension,  we  relied  on  an  abbreviation  of  Sheridan’s 
initial  autonomy  scale  which  folds  in  a  sense  of  where,  in  the  hierarchy  of  tasks  which  comprise 
the  mission,  the  control  is  taking  place: 


2 

Of  course,  determining  what  a  “conceptually  distinct  function”  or  task  type  is  will  be  subject  to  individual  judgment  and  to  the 
needs  and  focus  of  the  analysis.  This  is  largely  irrelevant  as  long  as  the  selected  level  is  kept  approximately  constant  across 
systems  to  be  compared. 
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1)  Human  is  in  charge  and  commands  specific,  limited,  non-integrated  functions  from 
automation 

2)  H  sets  overall  goals,  dictates  tasks,  but  delegates  moderate  decision  authority  within  isolated 
functions  to  A,  while  retaining  monitoring  and  intervention  authority 

3)  H  responsible  for  overall  goals,  but  A  is  given  large  tasks  which  may  integrate  across 
functions.  A  may  initiate  actions  within  its  functions 

4)  Balanced  responsibilities  between  H  and  A 

5)  As  for  3,  but  switch  H  and  A 

6)  As  for  2,  but  switch  H  and  A 

7)  As  for  1,  but  switch  H  and  A 

•  What’s  important  about  environmental  “Complexity”?  We  argued  that  this  could  be  captured  by  noting 
how  often  the  UMV  has  to  change  its  behaviors  (where  “behaviors”  are  significant  variations  within  the 
tasks  or  functions  defined  for  “Task  Diversity”  above).  This  dimension  was  called  “Behavioral  Change 
Frequency”  or  BFrq. 

•  Behavioral  Change  Frequency  (BFrq)  -  What  is  the  average  duration  between  required 
changes  in  vehicle  behaviors  (either  user-  or  system- initiated)  in  a  typical  mission  of  interest?: 

1)  Longer  than  1  per  hour 

2)  Every  20  -  60  min 

3)  Every  5-20  min 

4)  Every  1-5  min 

5)  Every  10  -  60  sec 

6)  Every  5-10  sec 

7)  Once  per  second  or  faster 

•  What’s  important  about  “Operator  Capabilities”?  Here,  we  felt  that  all  the  required  operator 
capabilities,  while  significant  in  their  own  right  for  training  and  selection,  etc.,  could  be  rolled  up  and 
reflected  in  how  many  operators  are  required  to  control  the  vehicle(s)  in  a  typical  mission  on  which 
the  analyst  is  focusing  -  hence,  Operator  to  Vehicle  Ratio  (Op). 

•  Operator  Vehicle  Ratio  (Op)  -  In  the  scale  developed  below,  ratio  can  be  calculated  -  thus,  four 
operators  controlling  four  UMVs  yields  a  ratio  of  1:1.  Similarly,  fractions  of  an  operator’s  time 
may  be  considered  if  the  operator  is  concurrently  engaged  in  other  tasks  -  thus,  an  infantry 
soldier  who  is  spending  half  his  time  controlling  and  monitoring  video  feed  from  a  UAV  while 
providing  covering  fire  could  be  represented  as  .5  operators  to  1  UMV  or  1:2. 

1)  4+  operators  to  1  UV 

2)  2-3  Ops  to  1  UV 

3)  1  to  1 

4)  1  Op  to  2  UVs 

5)  1  Op  to  3 -4  UVs 

6)  1  Op  to  5 -10  UVs 

7)  1  Op  to  10+ UVs 

•  What’s  important  about  “UMV  Capabilities”?  Here,  we  argued  the  raw  capabilities  of  the  UMV  were 
not  as  important  (and  were  too  diversified  for  good  abstraction  in  a  model),  but  rather  its  capabilities 
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to  perform  its  functions  without  operator  intervention.  This,  in  turn,  gave  rise  to  a  focus  on  the 
frequency  with  which  the  operator  had  to  intervene  in  the  functioning  of  the  UMV(s). 

•  Operator  Intervention  Frequency  (IFrq)  -  This  dimension  was  captured  in  a  scale  tied  to  the 
required  frequency  with  which  the  operator  had  to  interact  with  the  system  to  achieve  successful 
mission  behavior.  (Note  that  later  thinking,  not  adopted  at  the  time  of  presenting  and  discussing 
this  dimension,  suggests  that  this  should  not  be  a  simple  intervention  frequency,  but  rather  a 
percentage  or  ratio  of  clock  time  which  the  operator  must  spend  in  interaction  with  the  vehicle  - 
similar  to  “Robot  Attention  Demand”  in  Olsen  and  Goodrich’s  (2003)  “fan  out”  metric.): 

1)  Once  per  second  or  faster 

2)  Every  5 -10  sec 

3)  Every  10  -  60  sec 

4)  Every  1-5  min 

5)  Every  5-20  min 

6)  Every  20  -  60  min 

7)  Longer  than  1  per  hour 


Once  these  scales  had  been  developed,  we  sought  to  illustrate  them  on  a  set  of  examples  drawn  from 
significantly  different  UMV  systems.  We  chose  a  set  of  three  systems: 

1)  Unattended  Ground  Sensors  (UGSs)  -  These  represented  a  very  simple  (perhaps  degenerate)  example 
of  UMV  systems.  A  UGS  is  a  simple  sensor  which,  once  installed  (or  even  air  dropped),  transmits  a 
video  or  auditory  signal  only  when  a  stimulus  of  interest  (e.g.,  a  heavy  vehicle  passing  by)  is  detected. 
Tens  or  perhaps  hundreds  of  UGSs  can  be  installed  in  an  area  and  “controlled”  by  a  single  operator. 
Once  installed,  they  do  not  move  or  change  their  behaviors  except  transmitting  vs.  not  transmitting. 

2)  Raja fs  RoboFlag  -  In  2003  and  2004,  Dr.  Raja  Parasurman  and  others  worked  with  a  simulated  robotic 
platform  created  by  Dr.  Mark  Campbell  of  Cornell  University  to  conduct  a  series  of  experiments  in 
flexible,  delegation-style  supervisory  control  [11], [12], [13], [14].  In  this  testbed,  a  single  human  operator 
controlled  a  team  of  five  ground-based  robots  maneuvering  them  about  a  playing  field  to  play  a  game  of 
“capture  the  flag”  against  a  fully  automated  team  of  five  opposing  robots.  The  robots  could  move,  sense 
other  robots,  “tag”  other  robots  to  disable  them  and  grab  the  enemy’s  flag  and  return  it  to  their  territory. 
The  operator  could  control  these  robots  (in  varying  experimental  conditions)  by  either  individual 
waypoint  commands,  or  a  series  of  increasingly  aggregated  “plays”  which  might  task  a  single  robot  or 
the  entire  team. 

3)  Jay  Shively's  MUSIM  and  Delegation  Control  (DelCon)  Environment  (a.k.a.  “daybook”)  -  DelCon 
[15], [16]  is  a  flexible  delegation-style  supervisory  control  system  being  developed  by  Jay  Shively  and 
colleagues  at  the  U.S.  Army’s  Aeroflightdynamics  Directorate.  Deleon  (or,  as  it  was  affectionately 
known  by  members  of  the  working  group,  “Jaybook”)  is  embedded  in  the  Multiple  UAV  Simulation 
(MUSIM)  testbed,  where  it  controls  three  UAVs  in  the  performance  of  an  urban  target  monitoring, 
search,  tracking  and  prosecution  scenario.  Jaybook  provides  control  capabilities  that  range  from 
waypoint  control  and  joystick-controlled  sensor  operations  to  multiple  UAV  coordinated  monitoring 
and  lasing/prosecution  plays. 


Using  the  scales  defined  above,  these  three  example  systems  can  be  graphed  to  illustrate  the  characteristic 
differences  between  them.  Figure  2-8  shows  a  traditional  linear  graph  for  each  example  system.  Note  that  both 
the  RoboFlag  and  Jaybook  examples  define  regions  (rather  than  simple  lines)  because  they  can  be  operated  in 
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flexible  modes  and  those  modes  have  different  implications  for  dimensions  such  as  how  frequently  the 
operator  must  intervene  (Ifrq)  and  how  much  autonomy  is  afforded  to  the  system  (A). 
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Figure  2-8:  Linear  Graphs  of  Values  for  Each  of  the  Seven 
Dimensions  for  the  Three  Exemplar  Systems. 


These  linear  graphs  make  comparisons  of  the  different  examples  quick  and  easy.  It  is  obvious  that  Raja’s 
RoboFlag  and  Jaybook  are  both  capable  of  being  operated  in  different  modes,  while  UGSs  are  not. 
Furthermore,  it  is  obvious  that  UGSs  are  operated  in  a  much  longer  Typical  mission  (T)  with  many  more 
“Vehicles”  (VS)  and  with  better  (rarer)  Intervention  Frequency  (IFrq),  but  with  much  less  task  Diversity  (D) 
and  Behavioral  change  Frequency  (BFrq)  than  either  of  the  others. 


RTO-TR-HFM-170 


2-17 


FRAMEWORKS  FOR  SUPERVISORY  CONTROL 
OF  MULTIPLE  UNINHABITED  MILITARY  VEHICLES 


Perhaps  more  convenient  and  informative  still,  because  we  have  structured  the  dimensional  scales  to  be  of 
equivalent  lengths  and  orientations,  we  can  graph  them  using  a  polar  star  format  as  in  Figure  2-9.  Furthermore, 
the  polar  star  can  group  the  dimensions  in  interesting  categories,  for  example,  task  characteristics  (Duration  (T) 
and  Diversity  (D))  are  shaded  tan,  operator  characteristics  (Operator  Vehicle  ratio  (Op)  and  Intervention 
Frequency  (IFrq))  are  shaded  pink,  and  platform/vehicle  characteristics  (Behavior  Change  Frequency  (BFreq) 
and  Vehicles/Sub-systems  (VS))  are  shaded  green.  Here,  again,  characteristic  patterns  are  made  visible.  We  can 
easily  see  that  UGSs  are  very  “good”  for  the  operator  in  the  sense  that  they  require  rare  interventions  and  can 
support  a  high  operator-to-vehicle  ratio,  but  that  they  achieve  this  by  severely  restricting  task  diversity  and 
behavior  change  frequency. 
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Figure  2-9:  Polar  Star  Depictions  of  the  Values  for  Each  Example  System. 
Note  clustering  of  dimensions  into  Task  characteristics  (T  and  D  in  tan), 
Operator  Characteristics  (Op  and  IFrq  in  pink)  and  Platform  or 
Vehicle  Characteristics  (BFrq  and  VS  in  green). 


Strengths  and  Weaknesses 

When  initially  presented,  this  7D  model  was  reasonably  well  received.  It  was  seen  as  having  the  strengths  of 
characterizing  environment,  vehicle,  and  operator  characteristics,  as  well  as  the  supervisory  control  relationship 
between  them  -  and  of  doing  so  in  relevant  and  interesting  ways  at  a  reasonable  level  of  aggregation  for  the 
group.  It  was,  however,  also  seen  as  still  too  complex  for  easy  comprehension.  Furthermore,  there  was  a  feeling 
that  many  of  the  dimensions  were  either  poorly  defined  or  confounded  (non-orthogonal),  or  that  the  scales 
proposed  were  sub-optimal  in  some  way.  In  practice,  though,  we  began  work  on  a  simplified  version  of  this 
seven  dimensional  model  (as  described  next)  to  solve  the  complexity  problem  instead  of  concentrating  on 
refining  and  improving  the  dimensions  and  scales. 
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2.7  2D  INTERACTION  DESCRIPTION 

Given  the  feeling  that  even  the  7D  model  described  above  was  still  too  complex  for  our  purposes,  we  sought 
to  simplify  it  further.  Upon  thinking  more  deeply  about  the  situation,  we  felt  that  supervisory  control  is,  at  its 
root,  an  interaction  relationship  between  a  supervisor  and  one  or  more  subordinates.  Everything  else  is 
external  to  that  relationship  and,  while  it  is  not  incidental  or  unimportant,  it  certainly  increases  the  complexity 
of  a  description.  We  could  instead  focus  primarily  and  exclusively  on  the  nature  of  that  relationship. 
This  insight  led  us  to  the  two  dimensional  model  of  the  supervisory  control  interaction  and  relationship 
described  below.  Dr.  Miller  first  described  and  presented  this  model  at  the  HFM-170  meeting  in  Paris  in 
September  of  2009,  and  expanded  and  provided  examples  of  it  with  developed  scales  at  the  Dayton  meeting  in 
June  of  2010. 

After  an  analysis  of  various  supervisory  relationships  in  human-human  interactions  (see  Figure  2-10), 
we  concluded  that  they  could  be  reasonably,  and  usefully,  arrayed  along  two  dimensions: 

1)  The  attentional  demand  that  the  relationship  required  of  the  supervisory  in  order  to  accomplish  any 
useful  work;  and 

2)  The  “scope”  or  range  of  functions  and  capabilities  that  the  subordinate(s)  provide  for  performing 
useful  work  at  that  level  of  attentional  demand. 
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Figure  2-10:  Scored  Values  for  Attentional  Demand  and  Performance  Scope 
for  Each  of  13  Different  Human-Human  Supervisory  Control  Relationships. 
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At  the  time  this  initial  thought  exercise  was  performed,  we  had  not  developed  scales  for  these  dimensions, 
but  we  nevertheless  took  the  step  of  informally  rating  each  relationship  on  a  10  point  scale  ranging  from  low 
to  high3.  The  results  of  this  informal  exercise  are  shown  in  Figure  2-10. 

An  interesting  phenomena  emerged  from  these  ratings.  It  would  seem  that  when  the  dimensions  were  arrayed 
against  each  other  (see  Figure  2-11),  one  could  envision  a  “utility  horizon”  which  ran  roughly  along  the 
diagonal.  “Useful”  supervisory  control  relationships  are  those  for  which  the  cost  to  perform  useful  work  is, 
at  most,  no  more  than  the  usefulness  of  the  work  performed.  Since  the  attentional  demand  dimension  we  had 
identified  was  a  fairly  direct  measure  of  the  “cost”  to  the  supervisor  of  performing  the  work  and  the 
“performance  scope”  term  was  a  measure  of  the  range  of  useful  things  the  subordinate  could  perform,  it  served 
as  at  least  an  indirect  measure  for  the  benefit  or  usefulness  of  the  work.  Therefore,  intuitively,  relationships  that 
fell  on  or  below  the  diagonal  were  useful,  while  those  which  fell  above  the  diagonal  tended  to  be  less  useful. 
Such  relationships  could  also  be  characterized  somewhat  more  quantitatively  by  taking  the  ratio  of  the  demand 
score  to  the  scope  score  -  as  illustrated  in  the  rightmost  column  of  Figure  2-10.  Here,  higher  values  are 
indicative  of  less  “useful”  relationships  (from  the  perspective  of  performing  work  with  immediate  utility),  while 
lower  values  are  indicative  of  more  productive  relationships.  To  check  this  intuition,  we  also  provided  intuitive 
ratings  (on  a  five  point  scale  from  “-  -”  to  “+  +”)  and  then  calculated  a  Pearson’s  correlation  coefficient  between 
the  two  sets  (treating  the  second  scale  as  -2  to  +2).  The  resulting  value  was  -.765,  indicating  that  demand-to- 
scope  ratio  was  highly  negatively  correlated  with  our  sense  of  the  usefulness  of  the  relationship. 
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These  ratings  are  Dr.  Miller’s  alone. 
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The  results  of  this  initial  brainstorming  exercise  were  presented  at  the  Paris  meeting  in  2009,  and  were  met 
with  general  interest  as  a  potentially  promising  direction.  The  primary  criticism  was  that  both  the  dimensions 
and  the  scales  themselves  were  very  informally  defined.  Thus,  Dr.  Miller  took  the  step  of  trying  to  refine  and 
quantify  them,  as  described  below. 

Attentional  Demand  -  This  dimension  was  meant  to  capture  the  amount  and  frequency  of  time  and  attention 
required  by  the  supervisor  to  manage  the  system  and  achieve  the  work  desired.  Our  initial  thought  was  that 
Olsen  and  Goodrich’s  [17]  Fan  Out  metric  was  a  close  fit  to  what  was  needed.  Olsen  and  Goodrich  actually 
labeled  their  metric  “Robot  Attention  Demand”  (RAD)  and  defined  it  by  the  formula:  IE  /  (IE  +  NT),  where: 

•  IE  is  “Interaction  Effort”  -  the  time  (or  effort)  required  to  interact  with  the  robot;  and 

•  NT  is  “Neglect  Tolerance”  -  the  robot’s  effective  performance  time  without  intervention. 

•  We  presume  (though  this  is  not  clearly  stated  in  Olsen  and  Goodrich)  that  IE  +  NT  =  total  time. 

Thus,  RAD  is  the  proportion  of  time/effort  during  operation  that  the  supervisor  must  devote  to  interacting  with 
the  automation.  Based  on  RAD,  we  proposed  “SAD”  (System  Attentional  Demand)  -  the  proportion  of 
supervisor  time/effort  required  to  interact  with  the  system  in  order  to  perform  desired  work.  SAD  is  a  unitless 
metric  that  ranges  from  0  (for  completely  autonomous  automation  that  requires  no  supervisory  input)  to  1 
(no  effectively  “free”  human  time  -  the  supervisory  spends  100%  of  his/her  time  interacting  to  achieve  the 
desired  work). 

To  compare  multiple  systems  with  SAD,  it  is  important  to  maintain  consistent  assumptions  and  scoring. 
Important  considerations  include: 

•  Whether/what  “set  up  time”  to  include?  The  RAD  definition  was  unclear  (and,  in  fact,  has  been  criticized 
by  Crandall  and  Cummings  [18])  for  failing  to  consider  pre-mission  planning  and  configuration  time, 
as  well  as  engineering  and  design  time.  For  using  SAD  or  RAD  to  compare  multiple  systems,  any  set  of 
practices  with  regards  to  these  non-execution  time  parameters  may  be  used,  but  it  is  important  that  they 
are  applied  consistently  across  systems  that  are  analyzed. 

•  What  performance  context  assumptions  are  used?  Again,  when  assigning  time  and  effort  used  to  control 
and  task  the  automation,  it  is  also  important  to  maintain  consistency  in  assumptions  across  different 
systems  rated  if  the  goal  is  to  compare  those  systems.  For  example,  does  the  scenario  of  use  represent  a 
“sunny  day”  where  nothing  goes  wrong  or  a  worst  case  or  factored  error  assumptions,  etc.?  Is  the  user 
considered  a  novice,  an  expert  or  somewhere  in  between?  What  error  rates  are  assumed  for  the  user’s 
inputs?,  etc.  Again,  the  SAD  metric  can  accommodate  a  wide  range  of  different  assumptions,  but  it  is 
important  that  the  assumptions  be  applied  consistently  across  systems  rated. 

Performance  Scope  -  There  is  a  problem  with  using  only  SAD  or  RAD  as  the  basis  of  comparison  across 
supervisory  control  systems,  however.  Essentially,  both  compare  systems  only  on  the  percentage  of  supervisor 
time  they  require;  there  is  no  explicit  notion  of  the  level  of  system  effectiveness  or  work  accomplished  for  a 
given  level  of  SAD.  In  order  to  compare  system  functionality  or  effectiveness  using  SAD  requires  an 
assumption  of  homogenous  tasks  and  performance  targets  -  even  Olsen  and  Goodrich’s  [18]  Fan  Out 
application  of  RAD  presumed  a  homogenous  task:  “fanning  out”  a  set  of  robots  searching.  That  said,  there  is 
no  explicit  notion  of  the  domain  or  task  included  in  RAD  or  SAD.  Using  SAD  alone,  each  of  the  following 
examples  would  have  the  same  “attentional  demand”  value: 

•  Telling  a  fleet  of  100  UAVs  to  “stay  put”  on  the  tarmac  (that  is,  to  do  nothing); 

•  Telling  very  highly  autonomous  UAV  to  “execute”  it’s  trip  around  the  world;  and 
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•  Telling  an  efficient  secretary  to  plan  your  next  months’  trips  (assuming  s/he  already  has  access  to 
your  required  trips  and  times  and  knowledge  of  your  needs  and  preferences). 

In  each  of  these  cases,  the  supervisor’s  attentional  demand  is  one,  short  verbal  interaction.  On  the  other  hand, 
the  examples  differ  radically  in  the  scope  of  work  performed.  Hence,  we  felt  that  a  second  dimension  was 
needed  to  reflect  the  variety  of  tasks  or  functions  the  subordinate  automation  can  perform.  The  problem  is  that 
tasks  are  inherently  hierarchically  decomposable  and  characterizing  them  across  systems  and  domains  is 
notoriously  difficult.  Therefore,  in  order  to  maintain  some  consistency  in  comparing  different  applications, 
we  would  need  a  common  task  model  for  the  domain  of  interested  which  is  shared  by  the  applications/ 
relationships.  This  is  not  to  say  that  all  the  systems  must  perform  exactly  the  same  tasks  in  the  same  way, 
but  some  basis  for  comparison  across  tasks  was  necessary  -  otherwise  we  would  be  stuck  simply  saying  that 
the  systems  did  different  things.  One  way  to  accomplish  this  might  be  to  require  that  the  systems  all 
accomplish  a  shared  function  or  goal,  though  perhaps  used  different  methods  to  do  so. 

Given  such  a  model  (which  might,  necessarily,  be  fairly  abstract),  we  thought  we  could  perhaps  simply  count 
the  tasks  (at  a  given  level)  that  the  proposed  system  performs,  and  that  such  a  count  would  itself  provide  a 
metric  for  performance  scope.  The  worked  example  to  be  described  next  was  meant  as  a  thought  experiment 
to  test  whether  a  simple  count  of  the  tasks  performed  at  a  common  level  of  a  reference  model  could  serve  as  a 
reasonable  metric  for  performance  scope. 

An  Elevator  Example  -  To  test  this  hypothesis,  we  conducted  an  extended  thought  experiment  to  compare 
several  versions  of  a  supervisor/subordinate  system  (which  was,  in  most  cases,  also  a  human-automation 
system),  each  of  which  was  designed  to  perform  the  same  basic  function:  an  elevator  system  in  a  multi-story 
building.  A  “reference  model”  for  the  tasks  of  elevator  systems  might  be: 

1)  Summon/Initiate  -  call  a/the  elevator; 

2)  Select  Elevator  to  respond; 

3)  Move  to  Called  floor; 

4)  Control  Speed; 

5)  Position  Elevator  Vertically; 

6)  Open  Door; 

7)  Load  Passenger(s); 

8)  Select  Destination  Floor; 

9)  Close  Door; 

10)  Move  to  Destination  Floor; 

11)  Control  Speed; 

12)  Position  Elevator  Vertically; 

13)  Open  Door; 

14)  Unload  Passenger(s);  and 

15)  Close  Door. 

Note  that  this  is  intended  as  a  “spanning”  model.  Not  all  tasks  are  pertinent  or  performed  by  all  systems, 
and  not  always  in  this  order.  The  intent  is  that  alternate  elevator  systems  can  be  evaluated  on  whether  and  how 
they  perform  these  tasks  (with  what  mix  of  human  and  machine). 
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Next,  we  defined  a  set  of  human-elevator  systems  to  map  against  the  reference  model  drawn  from  the  variety 
of  elevator  systems  we  had  experienced: 

1)  Old  Style  -  Completely  Manual  Operation,  single  elevator.  In  this  style  of  elevator,  the  human 
(usually  a  dedicated  “elevator  operator”)  performed  door  opening/closing,  vertical  movement  and 
positioning,  etc. 

2)  Freight  Elevator  -  Here,  I  was  thinking  of  an  elevator  in  an  academic  building  at  the  University  of 
Chicago  where  a  button  press  controlled  opening/closing  the  door  but  the  human  controlled  the  rest 
(positioning  and  movement,  etc.). 

3)  English  “Moving  Carriage  ”  -  This  was  an  elevator  which  I  (and  others)  had  experienced  in  England 
-  where  elevator  “cars”  ran  on  a  continuous,  non-stopping  vertical  track,  there  were  no  doors  and 
riders  stepped  on/off  the  car  as  it  passed  by  the  opening  on  each  floor. 

4)  Current  Single  -  What  we’re  all  most  familiar  with:  a  modem  “automated”  elevator  typical  of 
moderate  sized  buildings.  A  button  press  summons  the  (single)  elevator  and  which  automatically 
opens  its  door  when  it  arrives  at  the  appropriate  floor  and  then  (usually)  automatically  closes  the  door 
when  people  board  or  leave.  A  different  button  is  pressed  for  each  floor  desired  and  the  elevator 
automatically  travels  to  that  floor,  positions  itself  and  opens  its  doors  for  riders  to  leave. 

5)  Current  Multiple  -  What’s  in  most  big  buildings,  hotels:  a  bank  of  elevators  for  different  floors/ 
regions.  A  single  button  is  pressed  to  summon  a  car,  but  the  automation  behind  the  bank  of  elevators 
controls  which  elevator  arrives  at  your  floor.  Riders  enter  and  push  buttons  for  their  desired  floor  and 
the  elevator  automatically  closes  its  doors  and  moves  to  the  desired  floor,  where  it  opens  its  doors  for 
disembarking. 

6)  New  York  Marriott  /  HFES  ’ 08  -  This  was  an  advanced,  optimized  bank  of  elevators  many  of  us 
encountered  at  the  Marriott  hotel  in  New  York  City  at  the  Human  Factors  and  Ergonomics  Society 
meeting  there  in  2008.  A  user  enters  the  desired  floor  in  a  central  console  and  is  told  which  elevator  to 
go  to.  The  elevator  arrives,  opens  its  doors  and  the  user  enters.  There  is  no  need  to  press  a  second 
button  to  indicate  the  desired  floor,  since  this  has  already  been  done.  Instead,  the  elevator  moves  to 
each  of  the  floors  users  have  indicated  they  want  to  go  to  -  and  supposedly  does  so  somewhat  more 
quickly  since  it  is  attempting  to  route  users  going  to  the  same  floors  into  a  common  car. 

Given  these  example  systems,  we  first  attempted  to  determine  a  SAD  metric  for  each  of  them.  This  was 
accomplished  by  estimating* 4  the  time  required  for  each  of  the  tasks  in  the  reference  model.  The  results  are 
presented  in  Figure  2-12.  Note  that  in  order  to  provide  a  comparable  number  across  the  systems  it  was 
necessary  to  assume  a  common  scenario.  We  chose  a  typical,  shared  task:  going  up  3  floors  as  a  single 
passenger.  Further,  we  noted  that  travel  speed  increases  with  more  modern  systems  and,  thus,  total  task  time 
(IE  +  NT)  decreases,  tending  to  drive  the  SAD  value  higher  than  it  would  otherwise  be5.  In  practice,  getting 
there  faster  enables  other  work  to  be  done  by  the  human  and,  thus,  perhaps  we  should  have  used  the  highest 


Note  that,  since  this  was  a  thought  experiment,  each  of  these  estimates  is  based  on  the  author’s  experience,  memory,  and  judgment, 

not  on  empirically  gathered  data. 

5  A  further,  hypothetical  system  will  illustrate  this  problem  more  dramatically:  imagine  a  teleportation  elevator  system  which 
requires  that  the  user  press  a  button  to  indicate  which  floor  s/he  wishes  to  go  to  and  then  instantaneously  transports  him/her  there. 
Such  a  system  would,  in  principle,  require,  say,  2  seconds  for  the  user  to  press  the  initial  button,  but  no  additional  time  to  get  to  the 
appropriate  floor.  Thus,  IE  would  be  2s  and  IE+NT  would  also  be  2s  and  SAD  would  be  2/2  =  1  -  a  value  we  associate  with  a  fully 
manual  system  above.  By  contrast,  if  we  took  the  IE  time  relative  to  the  total  time  for  the  worst  case,  most  manual  comparable 
system  (the  “old  style”  elevator),  we  would  have  2s  /  247s  =  .008  -  a  very  highly  automated  system.  This  seems  to  mesh  with 
intuition  more  neatly. 
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value  for  total  time  across  the  systems  which  perform  the  comparable  elevator  function.  This  insight  is  not 
reflected  in  the  values  in  Figure  2-12,  however. 


Alternate  Elevator  Systems 
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5s 
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2s 
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Os 

Os 
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210s 
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*  Extra  effort  beyond  continuous-  comnanded  movement 
**  Combined  with  the  scmmori/initlate  act 


Figure  2-12:  SAD  Estimates  for  6  Different  Elevator  Systems. 

We  then  created  scope  values  for  each  of  the  alternate  elevator  systems  by  simply  counting  which  of  the  tasks 
each  mechanical  (subordinate)  system  performed  automatically.  We  quickly  realized  that  many  systems 
partially  automated  some  of  the  tasks  and  we  chose  to  use  fractional  values  to  indicate  the  degree  to  which, 
in  the  scorer’s  judgment,  the  system  automated  the  task.  The  results  are  shown  in  Figure  2-13. 
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Alternate  Elevator  Systems 
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1 
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Unload  Passenger(s) 

0 

0 

0 

0 

.1 

.3 

15 
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0 

.8 

0 

.8 

.8 

.8 

TOTAL  SCORE 

.5 
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10.8 

12 

Percent  (out  of  15) 

3.3% 

24.7% 

30.7% 

66% 
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Figure  2-13:  Performance  Scope  Estimates  for  6  Different  Elevator  Systems. 

Armed  with  these  sets  of  quantitative  values,  we  were  now  able  to,  again,  graph  them  in  various  ways  to 
facilitate  interpretation.  Figure  2-14  shows  the  two  dimensions  plotted  against  each  other.  Interestingly,  in  this 
figure,  those  systems  which  are  clearly  less  fully  automated  cluster  in  the  upper  left,  while  those  which  are 
more  fully  automated  cluster  in  the  lower  right.  This  is  in  keeping  with  our  intuitions  that  the  more  modern 
systems  are,  in  fact,  better  representatives  of  “supervisory  control”  relationships  while  the  older  systems  are 
poor  examples  of  the  relationship.  This  suggests  that  the  diagonal  in  Figure  2-13  may  represent  a  rough 
definitional  boundary:  those  system  which  fall  above  it  are  not  “supervisory  control”  systems  precisely 
because  they  require  too  much  effort  from  the  human  supervisor  for  the  amount  (scope)  of  work  they 
accomplish.  By  contrast,  those  which  fall  below  the  line  are  good  examples  of  supervisory  control. 
The  “moving  carriage”  example,  which  falls  on  the  diagonal,  is  an  interestingly  ambiguous  case.  It  automates 
some  functions  but  still  requires  substantial  vigilance  and  attention  from  the  user  and  we  are  unsure  whether  to 
call  it  a  supervisory  control  system  or  not. 
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Figure  2-14:  Graphing  SAD  and  Performance  Scope  Dimensions  Against  Each  Other - 
With  a  Suggestion  of  a  Definitional  Boundary  for  “Supervisory  Control”. 


Figure  2-15  provides  a  slightly  richer  depiction  of  the  graph  by  characterizing  the  different  quadrants  of  it. 
Here,  we  might  be  able  to  assign  labels  to  suggest  the  kinds  of  relationships  which  characterize  the  systems 
which  fall  into  the  various  sections.  For  example,  the  upper  left  had  quadrant  is  characterized  by  comparatively 
high  attentional  demand  from  the  supervisor,  but  comparatively  low  scope  of  activities  which  the  subordinate 
can  perform.  We  might  label  relationships  in  this  quadrant  “child-like”  since,  like  interacting  with  a  child  or 
infant,  they  require  lots  of  supervision  in  order  to  perform  little  or  no  immediately  useful  work.  Relationships 
falling  into  the  upper  right  hand  quadrant  might  be  labeled  “teenager-like”  since,  like  interacting  with  a 
teenager,  substantial  supervision  is  still  required,  but  a  surprising  range  and  scope  of  work  can  be  accomplished 
if  a  supervisor  is  willing  to  take  the  time  required.  The  lower  left  hand  quadrant  might  be  characterized  as  like 
interacting  with  a  sheepdog  since  a  sheepdog  is  capable  of  performing  a  limited  range  of  behaviors,  but  can  do 
so  with  very  little  supervision  from  the  human  supervisor.  Finally,  the  lower  right  hand  quadrant  might  be 
characterized  as  like  an  “Awesome  Assistant”  (e.g.,  a  “Radar  O’Riley”  from  the  M*A*S*H  television  series) 
-  someone  who  has  a  very  wide  range  of  performance  capabilities  and  requires  little  supervision  to  perform 
them. 
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Figure  2-15:  Labeling  the  Quadrants  to  Convey  Intuitions  About  the 
Types  of  Relationships  Afforded  by  Systems  Which  Fall  Into  Them. 

Moving  Beyond  Elevators  -  The  above  thought  experiment  shows  promise  for  this  simplified  2D  model  since 
it  illustrates  the  model’s  ability  to  capture  interesting  differences  between  a  set  of  automation  systems  and  to 
mirror  our  intuitions  about  their  effectiveness  and  the  degree  to  which  they  exemplify  supervisory  control 
relationships.  Nevertheless,  we  realize  that  we  have  left  open  the  question  of  whether  or  not  this  framework 
will  prove  relevant  to  real-world  systems.  Easily  the  most  important  challenge  would  be  developing  an 
acceptable  “reference  model”  to  evaluate  performance  scope  for  a  set  of  realistic  UMVs.  While  we  did  not 
perform  this  task,  we  were  able  to  point  to  some  characteristics  of  potential  models  to  serve  as  starting  points: 

•  It  should  characterize  (and  decompose)  a  common,  shared  function  performed  or  goal  achieved  by  all 
systems  to  be  compared. 

•  Though  the  model  of  that  shared  function  can  be  fairly  abstract,  it  need  be  concrete  enough  to  support 
deriving  percentage  time  or  effort  estimates. 

•  It  is  helpful,  but  may  not  be  required,  if  there  are  shared  tasks  in  the  decomposition  of  the  shared 
function.  If  some  systems  require  a  sub-task  to  perform  the  function  and  others  do  not,  the  complete 
list  for  the  reference  model  can  include  the  union  of  all  of  the  tasks  and  scope  and  SAD  assessments 
can  indicate  whether  or  not  the  alternate  systems  perform  the  tasks  and  the  time  required. 

•  The  reference  model  may  need  to  be  augmented  by  a  specific,  shared  scenario  (again,  as  performed 
by  all  systems  to  be  compared)  to  enable  temporal  SAD  computations. 

While  we  did  not  develop  such  a  model  for  UMV  comparisons,  one  might  be  built  out  of  shared  vehicle 
functions  such  as  navigation,  propulsion,  sensing,  etc.  One  such  model  for  aviation  UMVs  might  be  derived 
from  typical  functions  of  aircraft  missions-  such  as  those  illustrated  in  the  “automation  trust”  pyramid  that 
Col.  Jeff  Eggers  of  the  U.S.  Air  Force  has  created  (see  Figure  2-16  for  Col.  Eggers  previously  unpublished 
model).  Col.  Eggers  uses  this  pyramid  to  convey  the  notion  that  trust  in  automation  must  be  built  from  the 
bottom  up,  but  it  also  serves  as  a  general  task  or  function  decomposition  for  typical  aircraft  missions.  It  is 
likely  that  this  model,  or  portions  thereof,  could  serve  as  the  basis  for  a  reference  model  for  at  least  UAVs  for 
performing  the  type  of  SAD  and  Performance  Scope  analysis  illustrated  for  elevators  above. 
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Figure  2-16:  Col.  Jeff  Eggers’  “Trust  Pyramid”  Which  Represents  a  Typical  Decomposition  of 
Aviation  Functions  and  Might  be  Useful  as  a  Reference  Model  at  Least  for  UAV  Comparisons. 


Strengths  and  Weaknesses 

There  was  general  consensus  that  this  2D  model  had  done  a  reasonable  job  of  operationalizing  and 
quantifying  the  two  dimensions  and  making  them  reusable  across  systems  and  applications  to  be  analyzed. 
Similarly,  this  model  has  the  strength  of  being  very  simple  to  explain  and  convey,  thereby  making  it  very 
suitable  for  use  as  an  organizing  framework  for  presenting  the  systems  from  this  working  group. 

On  the  other  hand,  it  was,  perhaps,  not  quite  as  general  as  would  be  ideal  due  to  the  need  for  a  shared 
reference  model  (which  would  necessarily  be  at  least  somewhat  task  and  domain  specific).  Since  we  did  not 
have  time  to  complete  investigating  the  development  of  a  reference  model  for  the  supervisory  control  systems 
under  investigation  by  the  HFM-170  members,  we  cannot  say  with  certainty  whether  a  single,  common 
reference  model  for  all  of  our  systems  is  possible.  Some  of  us  were,  in  fact,  sceptical  that  a  single  reference 
model  could  encompass  the  air,  sea,  and  ground  applications  being  investigated,  much  less  the  component 
systems  such  as  alternate  visualizations  or  control  systems  to  support  supervisory  control  systems. 

More  seriously,  though,  there  appeared  to  be  general  consensus  that  this  2D  model  may  have  gone  too  far  in 
simplifying  the  characterization  of  supervisory  control  relationships,  that  it  had  suppressed  too  much 
interesting  detail  between  the  alternate  systems.  Having  seen  the  results  of  this  2D  model  development, 
several  group  participants  were  interested  in  returning  to  (and  further  refining)  the  7D  model. 
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2.8  CONCLUSIONS 

Clearly,  this  ongoing  discussion  of  frameworks  for  supervisory  control  has  illustrated  that  a  very  wide 
diversity  of  such  models  are  possible,  each  with  different  strengths  and  weaknesses.  While  no  single  model 
emerged  with  which  to  present  the  results  of  this  workshop,  we  did  identify  several  dimensions  that  seem 
relevant  to  discriminating  between  supervisory  control  approaches  being  examined  by  this  group,  and  we 
proposed  methods  for  identifying  and  characterizing  supervisory  control  relationships  -  particularly  in  the 
2D  model  described  above. 

As  has  been  noted  before  us  (most  notably  by  Parasuraman,  Sheridan  and  Wickens,  [6]),  Sheridan’s  original 
model  of  levels  of  autonomy,  while  convenient,  confounds  many  dimensions  of  supervisory  control 
relationships  and,  ultimately,  does  not  give  a  good  sense  of  how  such  systems  operate  and  what  they  do. 
Several  alternate  models  have  been  proposed,  including  some  in  this  document  for  the  first  time,  which 
expand  and  refine  our  notion  of  supervisory  control  relationships  as  they  exist  in  alternate  systems.  More 
importantly,  these  models  have  different  strengths  and  weaknesses.  Some  are  very  detailed,  specific  and 
precise  -  but  that  very  precision  comes  at  a  cost  of  both  greater  effort  to  construct  representations  of  alternate 
systems  within  the  framework  and  greater  effort  to  understand  the  system  characterization  when  it  is  later 
presented.  Such  approaches  might  be  appropriate  for  design  and  evaluation  of  a  given  system  (or,  as  with 
Miller  and  Parasuraman,  [9],  for  conveying  specific  delegation  actions  to  automation),  but  they  are  not 
particularly  convenient  for  giving  a  “feel”  of  the  system  for  comparison  purposes.  That  said,  any  framework 
which  does  not  express  such  precise  details  will,  inevitably,  suppress  some  aspects  of  system  design  or 
operation. 

Our  examination  of  the  LoA3  model  (and,  to  a  lesser  degree,  the  2D  Interaction  Description  model)  showed 
that,  even  though  the  term  “supervisory  control”  arguably  defines  a  relationship  between  supervisory  and 
subordinate,  any  framework  which  concerns  itself  exclusively  with  this  relationship  and  does  not  concurrently 
capture  aspects  of  the  operator,  system  and  environment  or  task  domain  of  usage  is  likely  to  be  seen  as 
insufficient.  Instead,  frameworks  which  seek  to  provide  a  basis  for  comparing  and  representing  a  set  of 
alternate  systems  or  approaches  should  also  capture  aspects  of  the  equipment,  personnel  and  context  of  usage 
-  especially  when  those  aspects  vary  in  interesting  ways  from  system  to  system. 

Most  of  the  models  examined  in  the  working  group,  and  reported  in  this  document,  focused  on  the  tasks  or 
functions  to  be  performed  by  the  human  +  automation  system.  While  there  is  an  ongoing  debate  in  the  Human 
Factors  community  over  the  relative  strengths  and  weaknesses  of  prescriptive  task  analysis  vs.  ecological 
function  or  goal  analysis,  the  models  proposed  here  are  largely  agnostic  to  the  distinction.  They  are,  however, 
focused  on  allocation  of  functions  between  human  and  automation  in  some  fashion  -  whether  by  goal  or  state 
or  function  of  scripted  task.  We  believe  this  is  due  to  the  nature  of  supervisory  control  relationships  -  which 
were,  after  all,  the  focus  of  study.  Supervisors  necessarily  retain  some  functions  as  their  exclusive  purview, 
share  or  retain  others  dynamically  and  in  various  combinations,  and  rely  exclusively  on  their  subordinates  for 
performing  still  others. 

At  the  end  of  this  exercise,  we  believe  that  the  7D  model  held  the  most  promise  for  satisfying  the  ends  of  this 
working  group.  This  model  was  largely  descriptive,  but  it  captured  several  dimensions  relevant  to  the  alternate 
supervisory  control  systems,  relationships  and  usages  we  were  examining.  While  the  specific  dimensions 
examined  might  or  might  not  be  the  best  ones,  and  the  scales  for  characterizing  them  might  also  be  improved 
upon,  this  multi-dimensional  description  of  alternate  systems  seemed  to  provide  the  right  level  and  type  of 
information  for  rapidly  and  easily  conveying  to  ourselves  and  others  how  a  set  of  supervisory  control  systems 
are  similar  and  different  from  each  other. 
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3.1  DATES 

29  September  -  3  October  2008. 


3.2  LOCATIONS 

Faculty  of  Engineering,  Memorial  University  of  Newfoundland,  St.  John’s,  Newfoundland,  and  technical 
demonstration  on  Bell  Island,  Newfoundland,  Canada. 


3.3  SCENARIO/TASKS 

The  Concept  Of  Operation  (CONOP)  was  to  simulate  a  civilian  Ground-Control-Station  (GCS)  crew  as  the 
Unmanned  Aircraft  System  (UAS)  service  provider  and  a  military  crew  at  a  Forward  Control  Station  (FCS)  as 
the  client  for  the  data.  The  civilian  crew  would  be  responsible  for  operation  of  an  Unmanned  Aircraft  (UA)  in: 
take-off,  transition,  return-to-base  and  landing,  and  hand-off  the  control  of  sensor  payload  and  limited  UA 
maneuver  to  the  military  crew  once  the  UA  reached  the  target  area.  The  sensor  data  would  be  accessible  in 
real  time  to  the  military  FCS  crew. 


3.4  TECHNOLOGIES  EXPLORED 

In  the  context  of  multi-agent  supervisory  control  of  Unmanned  Vehicle  Systems  (UVS),  the  technology  matrix 
consists  of  the  following  cases: 

1)  A  single  crew  controlling  a  single  UVS; 

2)  A  single  crew  controlling  multiple  UVS  (force  multiplication); 

3)  Multiple  crews  controlling  a  single  UVS  (this  technology  demonstration);  and 

4)  Multiple  crews  controlling  multiple  UVS. 

Although  force  multiplication  (Case  2)  is  often  cited  as  the  ultimate  goal,  Case  4  is  a  more  realistic  objective 
because  a  UVS  is  a  complex  system,  and  often  involves  multiple  crews  in  its  operation.  The  CAN-1  technology 
demo  focuses  on  Case  3  as  a  precursor  to  the  implementation  of  Case  4.  The  multiple-crew  CONOP  in  Case  3  is 
an  example  of  multi-agent  supervisory  control:  the  GCS  provides  the  high-level  supervisory  task  of  bringing  the 
UA  from  the  launch  and  recovery  location  while  the  FCS  is  tasked  with  the  low-level  control  of  the  sensor 
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payload  while  the  UA  is  on-station.  This  CONOP  can  easily  be  extended  to  Case  4  by  using  the  GCS  crew 
dispatching  multiple  UA  to  different  FCS  at  different  locations. 

The  UA  often  has  to  transit  over  non-segregated  airspace  from  the  launch  and  recovery  site  to  the  on-station 
sites  [1],[2].  Sense  And  Avoid  (SAA)  technology  [3], [4]  is  needed  to  ensure  the  safe  integration  of  unmanned 
aircraft  with  other  manned  traffic  in  this  transit  over  the  non-segregated  airspace.  This  technology  demo  fits 
within  RAVEN  II,  a  research  and  development  program  conducted  by  Memorial  University  of  Newfoundland 
to  develop  SAA  technology  for  small  UA. 


3.5  HUMAN  FACTORS  ISSUES  EXPLORED 

The  RAVEN  group  is  interested  in  the  human  factors  issues  in  qualifying  situation  awareness,  responsibilities 
and  competence  of  each  crew  over  different  phases  of  the  UA  mission,  in  particular  with  respect  to  SAA 
responsibilities.  The  External  Pilot  (EP)  is  tasked  with  the  see-and-avoid  responsibilities  at  all  times  for  visual 
deconfliction  of  traffic  when  the  UA  is  within  visual  range  of  the  EP.  SAA  duties  are  assigned  differently  over 
the  three  mission  phases:  launch  and  recovery,  transit  and  on-station,  and  over  two  different  crews: 
the  civilian  GCS  crew  and  the  military  FCS  crew,  as  shown  in  Table  3-1.  Of  particular  interest  is  the  skill 
competence  for  the  external  pilots  at  the  GCS  and  at  the  FCS.  It  is  expected  that  the  EP  at  the  FCS  would  have 
limited  ability  to  tele-operate  the  UA,  and  his/her  duties  will  mostly  be  the  command  and  control  of  the  sensor 
payload,  and  to  prevent  the  UA  from  falling  into  the  possession  of  hostile  forces. 


Table  3-1:  GCS  and  FCS  Crew  Responsibilities  and  Competence. 


Mission  Phase 

Crew  Responsible  for 

SAA  Situation 

Skill  Level  of  the  EP 

SAA  Duties 

Awareness 

Launch  and  Recovery 

GCS 

Visual  +  Instrument 

High 

Transit 

GCS 

Instrument  Only 

NA 

On  Station 


FCS 


Visual 


Low 


3.6  UNMANNED  SYSTEMS  USED 

The  UA  was  an  Aerosonde  Mk  4.2  equipped  with  a  Piccolo  Plus  autopilot  from  Cloudcap  Technologies  and 
an  EO  sensor.  The  UA  was  launched  from  a  mobile  command  centre  equipped  with  two  completely  redundant 
GCS  units.  The  Piccolo  ground  control  station  software  (version  4.0.3)  and  stageboxes  hardware  were  used  in 
the  GCS  and  FCS  units. 


3.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

The  results  of  the  study  have  been  presented  at  the  Task  Group  meeting  following  the  demo.  The  following 
table  summarizes  the  extent  of  the  NATO  collaboration.  There  have  been  follow  up  collaborations  between 


3-2 


RTO-TR-HFM-1 70 


dMNATO 
WP  I  OTAN 


CAN-1:  MULTI-CREW  CONTROL  OF  A  SINGLE  UNMANNED  AIRCRAFT 


the  Canada  and  the  US  task  forces  on  the  training  requirement  for  the  EP.  There  are  also  on-going  efforts  in 
communications,  coordination  and  collaboration  between  Canada,  Germany  and  Portugal  on  the  planning, 
design,  execution  and  analysis  of  multiple  flights  tests  involving  small  unmanned  aircraft  near  or  over  the 
North  Atlantic  Ocean,  involving  possibly  beyond- visual-range  and/or  night-time  operations. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

X 

X 

Coordination 

X 

X 

X 

Collaboration 

X 

X 

X 

3.8  SUMMARY  OF  TD  RESULTS 

3.8.1  The  Planned  Demo 

The  Canadian  hosts’  plan  was  to  demonstrate  the  hand-off  of  control  of  the  UA  from  the  launch-and-recovery 
GCS  crew  to  a  FCS  crew.  Once  the  UA  reached  its  altitude  and  a  trimmed  flight  condition,  the  UA  would  be 
put  under  autopilot  mode  commanded  by  the  UVS  operators  inside  the  mobile  command  centre.  The  UA 
would  fly  a  fixed  pattern  overhead  flight  to  simulate  the  transit  from  the  launch  site  to  the  target  area.  After  a 
certain  time,  the  UA  was  assumed  to  have  reached  its  target  area.  Control  would  then  be  passed  off  to  a 
portable  ground  control  station  simulating  a  FCS  crew  located  near  the  target  area.  The  FCS  crew  would 
monitor  the  Electro-Optical  (EO)  imagery  and  could,  optionally  reprogram  the  flight  path  of  the  UVS  for 
additional  intelligence  gathering  over  target  area.  After  a  certain  time-on-station  period,  control  would  be 
passed  back  to  the  GCS  to  simulate  the  return  to  the  launch  area.  The  UAV  would  be  recovered  (landed)  by 
the  GCS  crew. 

3.8.2  The  Demo  Day 

On  the  afternoon  of  Wednesday  October  1,  2008,  members  of  the  NATO  HFM-078/170  team  shown  in  Figure 
3-1  witnessed  a  live  flight  demonstration  of  the  Aerosonde  Mk  4.2  UA  (named  “Takunnajik”  which  means 
“Seeker”  in  the  Innu  language  from  the  Canadian  North).  This  required  a  last-minute  determination  to  proceed 
based  upon  weather,  a  transit  to  the  ferry  terminal,  ferry  ride  over  to  Bell  Island,  and  transit  to  the  remote 
airstrip,  and  all  again  in  reverse. 
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Figure  3-1 :  NATO  Team  and  Aerosonde  UA  after  Landing. 


3.8.3  Hand-Off  Procedure 

The  normal  procedure  for  a  hand-off  from  GCS  to  FCS  is  to  have  the  GCS  operator  give  the  FCS  a  signal  for 
the  hand-off  while  the  UA  is  still  connected  to  the  GCS  via  a  command  and  control  communication  Channel 
(A).  The  FCS  is  set  up  on  a  new  communication  Channel  (B)  on  hot  standby.  The  GCS  operator  commands 
via  Channel  A  to  the  UA  to  switch  to  Channel  B  for  communicating  with  the  FCS.  If  the  UA  does  not  pick  up 
Channel  B  from  the  FCS  after  a  certain  timeout  period,  e.g.,  5  seconds,  the  UA  will  revert  back  to  Channel  A 
at  the  GCS.  Note  that  the  hand-off  is  bump-less  because  all  the  waypoints  are  stored  in  the  autopilot  on  the 
UA  and  the  UA  will  continue  its  mission  until  receiving  further  commands  from  the  FCS  after  the  hand-off. 
Also  note  that  the  UA  can  potentially  be  hijacked  by  a  hostile  FCS  if  the  hostile  FCS  emits  a  more  powerful 
signal  on  Channel  A  than  the  GCS  because  of  the  closer  proximity  of  the  FCS  to  the  UA.  This  vulnerability 
has  to  be  mitigated  via  a  secure  datalink. 

3.8.4  Actual  Events 

On  the  day  of  the  demo,  there  was  only  one  External  Pilot  (EP)  available  on  site,  and  it  was  deemed  to  be 
unsafe  if  control  was  handed  off  to  the  FCS  without  another  EP  as  a  safety  pilot.  It  was  decided  to  use  the 
second  redundant  GCS  as  shown  in  Figure  3-2  to  act  as  a  FCS.  The  console  on  the  right  was  the  GCS 
communicating  to  the  UA  on  Channel  A1  and  the  console  on  the  left  simulated  the  FCS,  communicating  to  the 
UA  on  Channel  B.  Both  consoles  were  located  within  the  mobile  command  vehicle. 


1  A  900  MHz  radio  link  was  used,  and  channel  designations  A  and  B  represents  different  numbered  channels  with  the  900  MHz 
band. 
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Figure  3-2:  Layout  of  GCS  and  FCS  in  the  Mobile  Command  Station. 


During  the  set-up,  there  was  Radio  Frequency  Interference  (RFI)  between  the  two  stageboxes  providing  the 
communication  links  between  the  two  GCS’s  and  the  single  UA.  It  was  decided  to  put  the  stagebox  on  the  left, 
simulating  the  FCS,  on  cold  standby  (turned  off).  After  the  UA  was  airborne  and  flying  autonomously  under 
autopilot  on  stored  waypoints  in  the  UA,  a  hand-off  was  attempted.  However,  a  temporary  link-loss  from  both 
GCS’s  occurred,  and  control  was  reverted  back  to  the  primary  GCS  automatically  once  the  five-second 
timeout  expired.  A  second  hand-off  was  successful  once  more  precise  timing  was  used  involving  turning  on 
the  FCS  stagebox  during  the  hand-off.  It  should  be  stressed  that  this  was  not  a  normal  or  correct  operating 
procedure  for  the  Piccolo  autopilot,  but  was  necessary  to  avoid  the  RFI  issue  caused  by  the  incorrect 
installation  of  two  Piccolo  GCS  stageboxes  in  close  proximity  to  each  other.  Later,  a  hand-off  from  the  FCS  to 
the  GCS  was  accomplished  successfully  without  needing  to  turn  on/off  any  of  the  stageboxes.  Following  this 
demo,  the  UA  was  landed  (Figure  3-1)  and  the  mission  was  completed. 


3.9  LESSONS  LEARNED 

The  first  lesson  pertained  to  the  skill  level  of  the  External  Pilot  (EP)  at  the  launch  site  near  the  GCS  and  on 
station  near  the  FCS.  The  GCS  software  used  was  not  STANAG  4865  compliant,  namely  that  both  the  crew  at 
the  GCS  and  at  FCG  have  the  full  control  of  the  autonomy  of  the  UA,  and  it  was  unsafe  to  leave  under  full 
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FCS  control  without  an  experience  EP  acting  as  the  safety  pilot.  Under  STANAG  4865,  the  FCS  might  only 
have  control  of  the  sensors  at  Level  1,  and  the  control  of  the  air  vehicle  would  have  remained  with  the  GCS, 
with  an  experienced  external  safety  pilot  as  in  the  case  of  the  visual-range  mission  as  in  the  Canadian  demo. 
From  Table  3-1,  the  EP  competency  was  high  for  the  GCS  crew,  especially  the  requirement  for  the  EP  to  be 
able  to  do  manual  landing  and  take-off  if  the  UA  did  not  have  Automatic  Take-Off  and  Landing  (ATOL) 
capabilities.  On  the  other  hand,  the  EP  competency  for  the  FCS  crew  should  be  low  since  information 
gathering  is  the  primary  task  and  not  UA  flying.  There  is  however  the  issue  of  flight  termination  when  the  UA 
was  on  station  under  the  FCS  control.  The  UA  could  be  damaged  or  hijacked  by  hostile  forces,  and  it  was 
important  the  UA  mission  can  be  altered  or  terminated  by  the  FCS  crew  to  prevent  the  UA  from  falling  into 
hostile  hands. 

The  second  lesson  was  spectrum  management.  The  problem  in  the  hand-off  was  peculiar  to  the  set-up  in  this 
demo:  The  FCS  was  located  next  to  the  GCS  causing  RFI  issues.  But,  the  general  issue  of  spectrum 
management  was  important.  The  RFI  issues  could  have  been  resolved  if  the  FCS  and  GCS  were  on  different 
frequency  bands:  900  MHz  and  2.4  GHz  as  in  the  Canadian  manufactured  Micropilot  system.  Another  issue 
was  the  danger  of  the  UA  being  hijacked  by  a  hostile  FCS.  This  would  be  an  important  topic  for  further 
research. 

The  last  lesson  was  on  the  proper  use  of  a  check  list.  The  last-minute  demo  was  compromised  by  not 
following  the  manufacturer’s  check  list.  It  was  known  that  two  Piccolo  stageboxes  should  not  be  located  in 
close  proximity  to  each  other  (under  2  meters).  This  contributed  to  the  RFI  issue.  If  the  checklist  had  been 
followed  and  the  mission  rehearsed  before  the  demo,  the  unsuccessful  first  hand-off  could  have  been  avoided. 


3.10  STUDY  CONSTRAINTS/LIMITATIONS 

3.10.1  Non-Segregated  Airspace 

The  flight  was  conducted  in  Class  G  non-segregated  airspace  in  close  proximity  to  the  St.  John’s  International 
airport.  Due  to  current  UVS  regulatory  restrictions,  the  entire  mission  was  done  at  visual  line  of  sight  distance 
from  the  manual  external  pilot. 

3.10.2  Limited  EP  Availability 

The  availability  of  a  single  EP  was  a  constraint  that  limited  the  full  implementation  of  an  FCS  with  another  EP 
at  a  different  location  than  the  GCS. 


3.11  CONCLUSIONS 

This  demonstration  marked  the  first  live-demonstration  of  unmanned  vehicle  supervisory  control  within  the 
NATO  HFM-078/170  Task  Group  experiences.  It  also  included  hand-off  demonstrations  between  two  UA 
supervisory  control  crews,  as  well  as  between  an  external  pilot  (flying  manual  control)  and  a  supervisory 
control  station.  The  flight  demo  was  well  received  by  all  and  sparked  many  interesting  crew  requirement 
discussions,  including  how  to  improve  upon  the  external  pilot’s  training/tasks.  Since  the  2008  demo, 
Dr.  O’Young’s  team  has  been  routinely  fielding  multi-UA  supervisory  control  flights  for  sense  and  avoid 
research. 
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3.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

Multi-agent  supervisory  control  of  multiple  UVS  can  be  formally  studied  in  the  context  of  a  multi-agent  hybrid 
system.  The  supervisory  tasks  at  the  GCS  can  be  modelled  as  discrete-event  [5]  tasks,  such  as  “change  of  flight 
plans”  and  “return  to  base”.  The  lower  level  task  at  the  FCS  can  be  considered  as  a  continuous  dynamical  task, 
such  as  the  manual  steering  of  a  camera  pointing  to  a  target.  The  interaction  between  the  discrete  tasks  at  the 
GCS  level  and  the  continuous  tasks  at  the  FCS  level  can  be  formalized  as  a  hybrid  system.  Hybrid  systems  [6] 
models  interactions  between  discrete,  e.g.,  decision  making,  and  continuous,  e.g.,  UA  dynamics,  processes 
within  a  unified  theoretical  framework.  The  application  of  hybrid  system  theory  to  UA  applications  have  been 
reported  in  [7], [8],  and  it  is  anticipated  that  a  formal  analysis  of  the  target  level  of  safety  of  an  SAA  system  can 
be  achievable  using  hybrid  system  as  an  underpinning  theoretical  foundation.  Future  collaborations  between 
Canada,  Germany  and  Portugal  could  provide  valuable  field  data  for  the  verification  of  this  theoretical 
framework. 
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Chapter  4  -  CAN-2:  BEHAVIOUR-BASED  COLLISION 
AVOIDANCE  AND  FORMATION  CONTROL 
OF  MULTIPLE  UNMANNED  VEHICLES 

Dr.  Bumsoo  Kim 

Defence  R&D  Canada  -  Ottawa 
3701  Carling  Avenue 
Ottawa,  Ontario  K1A  0Z4 
CANADA 

Email :  Bumsoo .kim@drdc-rddc . gc . ca 


4.1  DATES 

26-30  July  2010. 

4.2  LOCATION 

DRDC  Ottawa. 


4.3  SCENARIO/TASKS 

Multiple  UGVs  (Unmanned  Ground  Vehicles:  They  also  are  considered  as  simulated  UAVs)  to  reach  to  a 
destination  without  colliding  each  other  while  avoiding  obstacles). 


4.4  TECHNOLOGIES  EXPLORED 

In  this  study,  AIC  (Artificial  Impedance  Control)  is  applied  for  the  generation  of  trajectory  of  UGVs  instead 
of  pre-planning  the  trajectory.  AIC  is  a  Cartesian  space  control  and  is  one  of  the  control  techniques  which  can 
generate  trajectories  for  both  obstacle  free  and  obstacle  avoidance  cases  in  real  time.  One  of  the  advantages  of 
artificial  impedance  control  for  UGVs  motion  control  is  the  fact  that  it  enables  UGVs  to  perform  obstacle 
avoidance  tasks  without  knowing  the  full  geometry  of  the  obstacles  and  of  the  environment.  [1] 

In  the  present  study,  we  started  testing  AIC  algorithm  for  single  vehicle  trajectory  generation  and  obstacle 
avoidance  performance  using  simulation  and  experimentation. 

Then,  it  was  expanded  to  two  vehicles  reaching  to  the  designated  targets  while  avoiding  collision  with  each 
other  and  avoiding  obstacles  in  their  ways  to  targets. 

Thirdly,  a  five  vehicle  formation  control  and  single  target  oriented  behaviour-based  control  were  tested  in 
simulation. 
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4.5  HUMAN  FACTORS  ISSUES  EXPLORED 
4.5.1  Reference  Frame 

Figure  4-1  indicates  the  global  reference  frame  (T,Q,S)  that  will  be  used.  Since  the  X80  robot  only  supports 
planar  motion,  the  S  coordinate  will  often  be  ignored  in  this  work.  The  global  frame  is  fixed  with  heading 
defined  as  a  counter-clockwise  rotation  about  the  S  axis.  The  vehicle  will  use  a  body-frame  coordinate  scheme 
(Figure  4-2)  where  the  X-axis  is  always  pointing  forward  from  the  vehicle,  and  the  Y -axis  is  pointing  to  the 


left. 


Q  4 


Figure  4-1:  Global  Coordinate  System. 


H 


/ 


Figure  4-2:  X80  Local  Coordinate  System. 
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Transformations  from  body  coordinates  to  global  coordinates  can  be  done  as  follows: 


t  cos  #  -sin  #  0  x 
q  =  sin #  cos#  0  •  y 
#  0  0  1  # 


(1) 


4.5.2  Kinematic  Model 

The  goal  of  the  model  will  be  to  define  the  motion  for  all  points  on  the  platform,  for  a  given  set  of  known 
variables.  In  this  case,  the  speed  of  the  wheels  (Vi  and  V2)  and  the  length  /  from  the  wheels  to  the  center  of 
rotation  c,  are  known.  With  this,  we  will  describe  the  motion  of  the  center  of  rotation,  and  can  easily  define 
the  motion  for  all  other  points  from  there. 

Figure  4-3  shows  the  axle,  and  center  of  rotation  of  the  vehicle. 


< - ► 


Figure  4-3:  X80  Axle  and  Center  of  Rotation. 


Since  point  C  is  directly  in  the  middle  of  both  wheels,  its  forward  velocity  will  be  defined  by  half  of  the 
velocity  from  each  wheel.  Thus: 


vc  =-V,+-V2 


2  2 


(2) 


Assuming  there  is  no  side-slip  in  the  wheels,  we  can  also  assume  that: 


(3) 


For  the  vehicle’s  angular  rotation,  we  can  see  that  wheel  1  is  going  to  affect  the  rotation  negatively,  while  the 
2nd  wheel  will  have  a  positive  affect.  Fixing  one  of  the  wheels  while  driving  the  other  will  result  in  an  angular 
rotation  as  follows: 


V 


(4) 
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Summing  the  effects  from  both  wheels  will  give  the  platform  an  angular  rotation,  about  c,  of: 

r2-rt 


#„  = 


2/ 


(5) 


Combining  equations  1  -  5,  we  get  the  following  relationship  for  the  motion  at  c: 


f 

cos  <9 

-sin# 

0 

q 

= 

sin# 

cos# 

0 

[9 

0 

0 

1 

1/  1/ 

72  72 

0  0 

-1/  1/ 

.  72 1  721. 


7 


(6) 


4.5.3  Impedance  Controller 

Without  the  inherent  vehicle  dynamics,  the  impedance  controller  developed  here  is  actually  just  a  PD 
controller.  However,  some  attempts  have  been  made  to  simulate  the  dynamics  of  the  vehicle  and  as  such, 
we  will  continue  to  use  the  term  ‘impedance’. 

Figure  4-4  below  depicts  the  attractive  and  repulsive  forces  presented  on  the  vehicle  during  motion.  We  will 
denote  the  vehicle  as  M  and  the  goal  location  as  T.  Let  Rmb  be  the  distance  from  the  robot  to  the  closest 
obstacle,  rt  repulsive  force  field  radius,  F a  the  attractive  force,  and  F r  the  obstacle’s  repulsive  force. 


T 


Figure  4-4:  Impedance  Control  Force  Diagram. 
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The  repulsive  and  attractive  forces  are  calculated  as: 


R 


•O' 


mb 


Fa  =- 


Kn  -S 


-fbj  -Br-Vcur 

(V) 

■  V  VA  d  <  S 

cur 

(8) 

\/Ad>S 

where  K  and  B  are  the  controller  constants,  Ad  is  the  distance  to  the  goal,  and  S  is  a  constant  distance  after 
which  the  force  is  constant. 

Using  these  forces,  we  can  then  calculate  the  vehicle’s  desired  heading  by  summing  the  forces: 


Pes  =  tan 


-1 


f  F  a  ^ 


\Fr  J 


(9) 


To  steer  the  vehicle  to  the  desired  heading,  a  proportional  controller  was  used  to  determine  the  heading  rate: 


e=Kt-(ed-e,) 


(10) 


where  0curis  the  current  vehicle  heading.  Using  this  value,  along  with  the  desired  trajectory  speed  Vset, 
equations  2  and  5  can  be  used  to  calculate  the  individual  wheel  speeds  to  be  sent  to  the  platform. 

This  controller  will  cause  the  robot  to  move  with  constant  velocity  to  point  T7,  while  avoiding  any  obstacle 
along  its  path. 


4.5.4  Control  Block  Diagram 

Autonomous  navigation  was  implemented  for  a  single  robot  using  an  AIC.  An  attractive  virtual  force  pulls  the 
robot  to  its  goal,  while  a  repulsive  virtual  force  pushes  the  robot  away  from  obstacles.  The  magnitude  and 
direction  of  the  vector  sum  of  these  attractive  and  repulsive  forces  is  used  to  calculate  an  appropriate  velocity 
and  turning  rate  for  the  robot,  so  that  no  prior  path  planning  is  required.  The  block  diagram  showing  the 
general  flow  of  the  impedance  control  program  is  shown  in  Figure  4-5  and  Figure  4-6. 
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Figure  4-5:  Block  Diagram  of  the  Impedance  Control  Program. 


4.6  UNMANNED  SYSTEMS  USED 

4.6.1  Dr.  Robot  X80Pro 

Modified  X80Pro  [2]  is  a  WiFi  enabled  robot,  and  is  designed  for  use  as  an  autonomous  navigation  and 
control  research  platform.  It  comes  equipped  with  multiple  sensors,  and  low  level  motor  controllers,  enabling 
the  user  to  focus  solely  on  higher  level  algorithms.  An  SDK  is  also  available  for  the  windows  operating 
system,  simplifying  access  to  the  motor  drivers,  sensors,  and  communication  system.  However,  for  use  on 
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different  operating  systems,  the  raw  device  protocols  are  given  for  direct  integration.  For  Windows  use,  a 
detailed  description  of  the  SDK  and  how  to  get  up  and  running  with  the  X80Pro  platform  can  be  found  in  [2]. 

4.6.2  Player/Stage 

Player  [3]  is  a  network  server  for  robotic  control.  It  provides  access  to  a  platform’s  sensors  and  actuators  through 
well-defined  interfaces  over  a  TCP  connection.  As  such,  it  is  easy  to  set  up  any  type  of  network  topology 
provides  that  the  robot  and  associated  computers  are  connected  over  a  TCP  enabled  network. 

Stage  [4]  simulates  a  population  of  mobile  robots  and  sensors.  Supported  sensors  cover  most  areas  that  are 
used  within  the  robotics  community.  Player  can  access  the  actuators  and  sensors  in  the  Stage  simulation 
environment  in  the  same  way  that  it  would  the  actual  hardware.  As  such,  it  is  easy  to  simulate  new  algorithms 
and  then  transition  to  the  hardware  by  simply  changing  the  TCP  address.  Furthermore,  it  is  also  possible  to 
mix  both  simulation  and  hardware  environments.  An  experiment  could  be  set  up  where  the  sensors  are  read 
for  the  simulated  world  but  the  actuators  are  commanded  on  the  actual  hardware,  or  vice-verse.  The  options 
are  wide  ranging. 


4.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

Canada  (Bumsoo  Kim)  and  France  (Gilles  Coppin)  share  interests  in  ideas  and  technologies  with  regard  to  the 
swarming  concepts  of  multiple  autonomous  vehicles  operation.  Collaborating  in  this  area  of  research  is 
planned  by  establishing  joint  projects  and  by  seeking  opportunities  to  share  within  NATO  Nations  in  the  Task 
Group. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

X 

X 

Coordination 

Collaboration 

X 

4.8  SUMMARY  OF  TD  RESULTS 

Figure  4-7  shows  the  results  of  a  simulated  obstacle  avoidance  situation,  comparing  the  case  when  the  robot  is 
given  a  constant  velocity  and  the  case  when  the  robot  is  allowed  to  vary  its  speed  between  a  specified 
minimum  and  maximum.  Note  that  the  areas  where  the  red  data  points  are  more  densely  concentrated  indicate 
where  the  robot  slowed  down. 
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Figure  4-7:  Stage  Simulation  Results 
(Left:  Constant  Velocity  (PD  Control);  Right:  Variable  Velocity  (AIC)). 


For  simplicity,  only  one  repulsive  virtual  force  is  applied  to  the  robot  at  a  time.  In  the  initial  implementation, 
this  force  was  generated  from  the  obstacle  nearest  the  robot.  The  program  used  the  map  builder  module  to 
determine  the  coordinates  of  the  obstacle  closest  to  the  coordinates  of  the  robot,  and  the  force  was  then 
calculated  based  on  the  distance  between  them.  However,  there  are  certain  situations  in  which  the  repulsive 
force  is  ignored,  namely  when  the  obstacle  is  on  the  other  side  of  the  goal  from  the  robot,  or  when  the  obstacle 
is  behind  the  robot.  In  the  latter  situation,  a  problem  would  sometimes  arise  with  this  implementation.  That  is, 
even  if  there  was  an  obstacle  within  sensor  range  in  front  of  the  robot,  no  repulsive  force  would  be  applied  if  a 
closer  obstacle  happens  to  be  detected  behind  the  robot.  In  such  cases  the  robot  sometimes  had  a  delayed 
reaction  to  the  obstacles  in  front  of  it;  it  would  continue  on  a  straight  path  until  the  distance  to  the  obstacle  in 
front  was  less  than  the  distance  to  the  obstacle  left  behind. 

To  fix  this  problem,  the  map  builder  function  to  return  the  closest  obstacle  was  modified  to  also  take  into 
account  the  heading  of  the  robot,  so  that  it  returns  the  closest  obstacle  in  front  of  the  robot  (with  a  180° 
perspective).  After  this  change  was  implemented,  the  robot  became  more  responsive  to  the  objects  in  front  of 
it,  and  it  allowed  for  a  smoother  motion.  A  comparison  of  the  results  from  before  and  after  this  change  is 
shown  in  Figure  4-8  and  Figure  4-9  below. 
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Figure  4-8:  Simulation  Results  (Left:  360°  Obstacle  Return;  Right:  180°  Obstacle  Return). 


Figure  4-9:  Experimental  Results  (Left:  360°  Obstacle  Return;  Right:  180°  Obstacle  Return). 


4.8.1  Singularities 

There  is  a  special  case  in  which  another  problem  arises  with  the  impedance  control  program.  It  happens  when 
the  attractive  and  repulsive  forces  are  perfectly  lined  up  (for  instance  when  the  goal  is  on  the  opposite  side  of 
an  obstacle  from  the  robot).  In  the  absence  of  a  lateral  repulsive  force  to  tell  the  robot  to  try  to  go  around  the 
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obstacle,  it  will  travel  in  a  straight  path  until  it  gets  stuck  or  crashes  into  the  obstacle.  This  singularity  problem 
has  not  yet  been  overcome. 

4.8.2  Multiple  Robots 

Once  the  AIC  for  a  single  robot  was  developed  and  successfully  demonstrated  in  both  simulation  and 
experiment,  the  next  step  was  to  extend  that  functionality  to  multiple  robots.  The  first  test  that  was  done  was  a 
Stage  simulation  that  was  populated  by  three  independent  robots.  Each  robot  used  the  AIC  to  navigate  and 
each  was  given  a  separate  goal  point.  There  was  no  communication  between  robots;  they  could  only  detect 
each  other  as  obstacles  using  their  equipped  sonar  and  infrared  sensors.  Since  the  robots  were  essentially 
moving  obstacles,  it  was  not  only  important  for  the  robots  to  successfully  detect  each  other,  but  also  important 
for  the  robots  to  be  able  to  detect  when  the  others  had  moved  out  of  the  way.  Without  this,  the  map  builder 
would  continuously  populate  the  occupancy  grid  with  a  streak  of  obstacles  as  the  robots  moved,  and  it  would 
make  navigating  to  the  goals  impossible.  Figure  4-10  shows  the  results  of  two  such  simulations  -  one  with 
constant  velocity,  and  one  with  variable  velocity. 


Figure  4-10:  Stage  Simulation  Results  (Left:  Constant  Velocity;  Right:  Variable  Velocity). 

It  is  more  useful,  however,  for  robots  to  be  able  to  communicate  and  work  together  to  achieve  a  common  goal. 
Two  different  approaches  to  multiple  robot  control  were  tried:  a  neighbour-follower  approach,  and  a  behaviour- 
based  approach. 

4.8.3  Neighbour-Follower  Approach  (Formation  Control) 

The  main  goal  of  the  neighbour-follower  approach  is  for  the  robots  to  achieve  a  specified  formation  on  their 
way  an  end  point.  Initially  each  robot  is  assigned  a  ‘neighbour’  robot,  and  it  is  told  to  maintain  a  certain 
relative  position  with  respect  to  its  neighbour.  This  is  done  using  another  virtual  force,  called  the  formation 
force,  which  is  added  to  the  vector  sum  of  the  attractive  force  pulling  the  robot  to  the  goal  and  the  repulsive 
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force  pushing  it  away  from  obstacles.  In  order  to  calculate  this  formation  force,  in  the  absence  of  more 
advanced  sensor  equipment,  it  is  necessary  for  the  robots  to  communicate  their  positions  and  velocities  to  their 
followers.  The  following  equation  shows  how  the  formation  force  is  calculated  for  each  robot. 

Ff  =  Kf -{Ard  - Arc)-Bf -(Vcur -Vn)  (11) 

where  A rd  is  the  desired  position  of  the  robot  relative  to  its  neighbour,  A rc  is  the  actual  current  position  of 

the  robot  relative  to  its  neighbour,  V CUr  is  the  current  velocity  of  the  robot,  V  n  is  the  current  velocity  of  its 
neighbour,  and  Kf  and  Bf  are  controller  constants. 

This  implementation  was  initially  tested  with  two  robots  in  a  Stage  simulation.  The  robots  were  instructed  to 
form  a  horizontal  line  (1  m  apart)  and  move  to  a  goal  several  metres  away,  with  no  obstacles  obstructing  their 
path.  The  simulation  is  shown  in  Figure  4-11.  The  results  show  that  the  robots  do  indeed  achieve  the 
formation  relatively  quickly,  but  once  they  approach  the  end  point,  they  get  confused.  The  problem  was  that 
the  robots  were  both  given  the  exact  same  goal  point,  so  while  they  were  ‘fighting’  for  it,  they  were  unable  to 
maintain  the  formation.  This  problem  was  solved  by  giving  each  robot  a  separate  goal  at  a  relative  distance 
based  on  its  relative  position  in  the  formation.  Different  gains  Kf  and  Bf  were  tested  to  try  to  reduce  the 
oscillations  of  the  robots  when  they  were  getting  in  formation. 

5  r 
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Figure  4-11:  Simulation  Results  for  Two  Robots  Attempting  a  Horizontal  Formation. 


RTO-TR-HFM-170 


4-11 


CAN-2:  BEHAVIOUR-BASED  COLLISION  AVOIDANCE 

AND  FORMATION  CONTROL  OF  MULTIPLE  UNMANNED  VEHICLES 


4.8.4  Behaviour-Based  Approach  (Flocking  Control) 

A  behavior-based  approach  to  multi-robot  control  was  also  investigated.  In  this  method,  no  specific  formation 
is  explicitly  assigned  to  the  robots.  Instead,  each  robot  tries  to  maintain  a  certain  distance  (although  no 
specific  orientation)  with  respect  to  the  other  robots  in  its  vicinity.  For  example,  if  the  desired  distance 
between  robots  is  0.7  m,  then  robots  under  this  distance  from  each  other  will  experience  a  repulsive  force, 
while  robots  above  this  distance  will  be  attracted,  up  to  a  maximum  distance  of  1  m.  Robots  farther  than  1  m 
apart  will  ignore  each  other. 

When  the  X80  Pro  robots  only  made  use  of  their  sonar  and  infrared  sensors,  communication  of  positions  was 
required  in  order  to  distinguish  robots  from  obstacles.  In  this  case,  however,  instead  of  only  needing  to  know 
the  position  of  its  one  neighbour,  each  robot  required  the  positions  of  every  other  robot.  It  then  had  to 
calculate  its  distance  to  every  other  robot,  as  well  as  a  force  for  every  robot  in  range.  As  more  robots  were 
added  to  the  simulation,  the  computer  would  get  increasingly  bogged  down,  and  as  a  result  the  sampling 
frequency  of  each  robot  diminished. 

In  order  to  determine  the  extent  to  which  the  sampling  rate  had  an  effect  on  the  performance  of  the  robots, 
a  simulation  was  set  up  in  stage  involving  five  robots.  The  first  run  was  done  at  normal  simulation  speed 
(real  time),  and  it  was  determined  that  the  frequency  of  each  robot  was  approximately  1  Hz.  Another  run  was 
done,  this  time  at  a  slower  simulation  speed  (0.3  times  real  time),  which  allowed  more  time  for  the 
computations  to  be  completed,  effectively  increasing  the  frequency  of  the  robots  to  9  Hz.  The  results  in  Figure 
4-12  clearly  show  that  the  sampling  frequency  plays  an  important  role  in  the  stability  of  the  robots. 


Figure  4-12:  Simulation  of  Behaviour-Based  Control  (Left:  Frequency  1  Hz;  Right:  Frequency  9  Hz). 
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4.9  LESSONS  LEARNED 

Communication  link  is  very  important  to  the  control  of  multiple  unmanned  vehicles.  Distributed  computing 
power  is  essential  for  the  system  stability.  AIC  proved  to  be  an  effective  method  to  generate  trajectory,  avoid 
obstacle,  and  avoid  collision  with  other  Unmanned  Vehicles  (UVs). 

The  AIC  enables  UVs  to  avoid  obstacles  without  knowing  the  full  geometric  description  which  usually 
requires  a  complex  vision  system.  The  only  information  needed  is  the  closest  point  of  surface  of  an  obstacle 
from  the  vehicle  at  each  time  provided  by  simple  range  sensors. 


4.10  STUDY  CONSTRAINTS/LIMITATIONS 

Continuation  of  the  research  after  the  completion  of  present  project  is  in  question.  Pending  funding  opportunities 
proposals  are  submitted  for  further  investigation.  The  technology  is  very  high. 


4.11  CONCLUSIONS 

Advancements  of  the  state  of  the  art  in  supervisory  control  of  multiple  autonomous  vehicles  can  be  pursued  by 
studying  human  interface  aspects  and  the  basic  self-organizing  and  protecting  autonomous  control.  We  studied 
and  demonstrated  the  self-organization  and  protection  capabilities  of  multiple  autonomous  ground  vehicles 
simulating  air  vehicles  using  computer  simulations  and  verifying  the  results  with  experimental  platforms. 
The  Artificial  Impedance  Control  for  local  autonomy  including  collision  avoidance  and  trajectory  generation 
shows  excellent  results.  It  is  also  expanded  to  study  formation  control  and  flocking  control.  The  computer 
simulation  results  are  really  promising. 


4.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

The  operator  friendly  and  robust  ground  control  station  interface  should  be  researched  and  developed  for  the 
operational  capability  of  the  developed  technology.  And  autonomous  mission  management  and  more  robust 
flocking  control  algorithms  development  should  be  pursued. 
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5.1  DATES 

2009-2012. 

5.2  LOCATION 

Toronto,  Ontario,  Canada. 


5.3  SCENARIO/TASKS 

Supervisory  control. 


5.4  TECHNOLOGIES  EXPLORED 

Intelligent  Adaptive  Agents  (IAI)  to  manage  a  multi-modal  display  in  the  Ground  Control  Station  (GCS) 
interface  for  supervisory  control  of  an  Unmanned  Aerial  Vehicle  (UAV). 


5.5  HUMAN  FACTORS  ISSUES  EXPLORED 

5.5.1  Background 

Our  Technology  Demonstration  (Tech  Demo)  is  called  OmniSense.  It  is  currently  being  designed  and  developed 
to  demonstrate  the  efficacy  of  a  multi-modal  display  (i.e.,  the  presentation  of  visual,  auditory,  and  tactile 
information)  [1]  for  enhancing  supervisory  control  of  an  automated  UAV.  As  a  prelude  to  OmniSense’s 
theoretical  underpinnings,  we  will  initially  discuss  our  research  on  Intelligent  Adaptive  Interface  (IAI)  which 
will  set  the  stage  for  our  discussion  of  OmniSense. 

Hou  and  his  colleagues  [2], [3], [4], [5]  designed  and  developed  an  IAI  conceptual  framework.  An  IAI  is  an 
operator  interface  that  dynamically  changes  the  display  and/or  control  characteristics  of  human-machine 
systems  to  adaptively  react  to  external  events  in  real  time.  A  typical  IAI  is  driven  by  intelligent  software 
agents  that  help  satisfy  the  decision-making  and  action  requirements  of  operators  under  different  levels  of 
workload  and  task  complexity  by  presenting  the  right  information  or  action  sequence  proposals  or  by 
performing  actions,  in  the  right  format  and  at  the  right  time  [2], [5], [6]. 
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The  IAI  concept  was  investigated  within  a  multi-UAV  control  context.  The  selected  scenario  involved  UAV 
operations  in  support  of  counter-terrorist  activities.  The  IAI  was  developed  as  part  of  the  UAV  tactical  control 
stations  for  a  modernized  Canadian  Maritime  Patrol  Aircraft  CP- 140.  This  work  was  divided  into  three 
phases. 

In  the  first  phase,  the  IAI  concept  was  developed  [7].  Figure  5-1  shows  the  IAI  conceptual  framework,  which 
became  the  guidance  for  the  design  of  the  UAV  GCS  used  for  this  project.  A  generic  IAI  framework  has  four 
components  that  are  listed  below: 

•  Situation  Assessment  and  Support  System:  This  component  provides  information  about  the  objective 
state  of  the  aircraft/vehicle/system  within  the  context  of  a  specific  mission,  and  uses  a  knowledge- 
based  system  to  evaluate  the  situation;  this  information  is  then  provided  to  the  Adaptation  Engine 
component  of  the  IAI  system. 

•  Operator  State  Assessment:  This  component  provides  information  about  the  objective  and  subjective 
state  of  the  operator  within  the  context  of  a  specific  mission  relating  to  real-time  analysis  of  his  or  her 
psychological,  physiological  and/or  behavioural  state  (e.g.,  continuous  monitoring  of  workload, 
inferences  about  current  attentional  focus,  ongoing  cognition,  visual  and  verbal  processing  load), 
and  intentions  using  extensive  a  priori  operator  knowledge  (e.g.,  models  of  human  cognition,  control 
abilities,  and  communication). 

•  Adaptation  Engine:  This  component  utilizes  the  higher-order  outputs  from  the  Operator  State 
Assessment  and  Situation  Assessment  systems,  as  well  as  other  relevant  aircraft/vehicle/system  data 
sources,  to  maximize  the  match  between  aircraft/vehicle/system  state,  operator  state,  and  the  tactical 
assessments  provided  by  the  Situation  Assessment  system. 

•  Operator  Machine  Interface  (OMI):  This  component  provides  the  means  by  which  the  operator 
interacts  with  the  aircraft/vehicle/system  to  satisfy  mission  tasks  and  goals.  This  is  also  the  means  by 
which,  if  applicable,  the  operator  interacts  with  the  intelligent  adaptive  system  (e.g.,  a  tasking  interface 
manager). 
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*  Knowledge  of  mission  plans/goals 

*  Knowledge  of  mission  time-lines 

•  Knowledge  of  mission  tasks/activities 

•  Knowledge  of  mission  environment 


•  OMI  Design  Guidelines 

•  HCI  Principles 


*  Mission  Plan/Goal  Monitoring 

*  Control  (task/activities)  Monitoring 

*  Sensor  (environment)  Monitoring 


I 


OPERATOR 

MACHINE 

INTERFACE 


Automate  /  Aid 

\ 


SITUATION 

ASSESSMENT 


OPERATOR 
STATE 
ASSESSMENT 


•  Psychophysiological  Monitoring 

•  Behavioural  Monitoring 

•  OMI  Design  Guidelines 

•  Automation-design  Principles 


*  Model  of  human  cognition 

*  Model  of  human  control  abilities 

*  Model  of  human  communication 

*  Model  of  human  interaction 


Figure  5-1:  Conceptual  IAI  Architecture. 


The  IAI  framework  is  a  closed-loop  system  in  which  a  feedback  loop  re-samples  operator  state  and  situation 
assessment  following  the  adaptation  of  the  OMI  and/or  automation.  The  goal  is  to  adjust  the  level  of  adaptation 
so  that  optimal  operator  states  (e.g.,  performance  and  workload)  are  attained  and  maintained.  Based  on  this 
framework,  a  methodology  was  produced  to  analyze  UAV  operations  in  a  counter-terrorist  mission  scenario. 
The  scenario  reflected  a  portion  of  the  2004  Canadian  Forces  (CF)  Atlantic  Littoral  ISR  Experiment  (ALIX)  that 
employed  a  Medium-Altitude,  Long-Endurance  (MALE)  UAV  and  a  variety  of  other  sensors  in  a  littoral 
environment  using  domestic  security  and  peace  support  scenarios  [8].  The  results  from  the  ALIX  experiment 
were  used  to  develop  a  human-machine  task  network  model  that  was  then  implemented  in  an  Integrated 
Performance  Modelling  Environment  (IPME)  [9].  The  model  has  two  modes  for  controlling  multiple  UAVs: 

1)  A  conventional  interface  (i.e.,  without  an  IAI)  to  control  multiple  UAVs;  and 

2)  An  interface  with  IAI  automation. 

The  difference  between  mission  activities  with  and  without  IAI  aiding  was  reflected  in  the  time  taken  to 
complete  critical  task  sequences  and  task  conflict  frequency.  The  simulation  showed  that  the  use  of  an 
interface  with  the  IAI  mode  permitted  operators  to  complete  critical  task  sequences  in  reduced  time,  even 
under  high  time  pressure  [2],[10]. 
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The  second  phase  focused  on  the  design  and  implementation  of  IAI  prototype  interfaces  that  incorporated  six 
system  function  groups:  inter-crew  communications,  route  planning,  route  following,  screen  management, 
data-link  monitoring,  and  UAV  sensor  selection.  A  synthetic  environment  was  created  that  followed  the  North 
Atlantic  Treaty  Organization  (NATO)  Standardization  Agreement  (STANAG)  4586  interface  software  protocol. 
The  experimental  environment  had  three  control  stations  replicating  CF  CP- 140  tactical  compartment 
workstations,  with  a  set  of  displays  and  controls  for  each  of  the  UAV  crew  members:  UAV  pilot,  sensor  operator, 
and  tactical  navigator  (Figure  5-2).  The  experimental  environment  also  had  an  integrated  video  and  audio  data 
collection  suite  to  facilitate  empirical  assessment  of  IAI  concepts. 


Figure  5-2:  IAI  Experimental  Environment. 


Human-in-the-loop  experiments  were  conducted  in  the  third  phase  to  examine  operator  workload  and  interface 
adaptability  with  mock-up  UAV  control  stations.  Eight  crews  (24  operational  CP- 140  members)  participated 
in  the  experiment.  Each  crew  completed  a  two-day  experiment  that  assessed  operator  interfaces  with  and 
without  IAI  aiding.  The  results  showed  reduced  completion  time  for  critical  task  sequences  in  the  IAI  mode. 
Also,  there  was  a  significant  reduction  in  workload  and  an  improvement  in  Situation  Awareness  (SA)  [3], [4], [5]. 

The  OMI  component  of  the  IAI  developed  by  Hou  and  his  colleagues  [2], [3], [4], [5]  presents  information  only  in 
the  visual  modality.  In  UAV  operations,  an  abundance  of  information  is  presented  in  the  visual  modality, 
resulting  in  cognitive  overload  and  low  situation  awareness  during  periods  of  high  task  complexity  and  leading 
to  performance  degradation.  Multiple-resource  theory  suggests  that  offloading  information  from  overtaxed 
sensory  modalities  to  other  modalities  can  reduce  workload  [11].  The  effective  presentation  of  multi-modal 
information  in  the  non-dominant  modalities  of  hearing  and  touch  can  likely  enhance  the  perception  of 
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information  in  the  dominant  sensory  modality  of  vision  via  redundancy  and  complementary  information 
presentation  [1].  For  example,  when  the  same  information  is  mapped  to  multiple  modalities,  redundancy  gains 
such  as  faster  response  times  to  an  incident  are  observed  [12].  Also,  multi-modal  displays  can  increase  the 
bandwidth  of  information  transfer  [1].  Studies  that  examine  methods  to  offload  the  visual  modality  in  UAV 
applications  have  been  investigated.  Enhanced  UAV  monitoring  performance  was  observed  via  a  multi-modal 
display  [13], [14], [15], [16].  For  example,  Calhoun  et  al.  [15]  found  that  a  unique  redundant  alert  for  critical 
warnings,  whether  aural  or  tactile,  helped  participants  differentiate  warning  types  and  improved  reaction  time  to 
critical  events,  while  participants  performed  multiple  tasks  in  a  simulated  UAV  control  station. 

Designers  need  to  capitalize  on  the  benefits  of  multimodal  displays  that  would  lead  to  effective  operator 
decision  making.  This  is  a  challenging  task  [17].  Unlike  visual  displays,  the  mapping  of  information  to 
non-visual  displays  is  not  well  understood.  To  date,  only  a  few  studies  have  explored  mapping  techniques  for 
representing  information  in  auditory  displays,  e.g.,  [18].  Tactile  displays  are  becoming  increasingly  common 
and  much  has  been  learned  regarding  the  use  of  tactile  cuing  in  display  design  [19].  However,  most  tactile 
displays  appear  to  be  designed  in  an  ad  hoc  fashion  [20],  and  we  are  unaware  of  any  literature  that  has  tried  to 
describe  how  to  systematically  map  information  to  tactile  displays.  To  address  this  problem,  we  are  currently 
carrying  out  initial  work  that  would  lead  to  the  development  of  techniques  to  map  auditory  and  tactile 
information  systematically  in  the  OMI  component  of  the  IAI  framework.  This  framework  will  be  used  for 
providing  information  on  system  faults  and  environmental  hazards  in  the  supervisory  control  of  a  UAV.  In  our 
present  work,  system  faults  can  include  a  low  or  high  engine  Revolutions  Per  Minute  (RPM)  warning. 
Environmental  hazards  can  include  wind  shear  or  turbulence.  System  faults  and  environmental  hazards  will  be 
collectively  referred  to  as  critical  events.  The  IAI  framework  is  first  presented  before  describing  the  tech 
demo,  OmniSense,  which  is  a  simulated  UAV  GCS  multi-modal  display. 

5.5.2  OmniSense 

OmniSense  focuses  on  the  OMI  component  of  the  IAI  framework  and  introduces  the  concept  of  a  multi-modal 
display  to  the  OMI.  In  the  multi-modal  interface  of  OmniSense,  an  auditory  and  a  tactile  display  will  be  used 
to  present  specific  display  variables  to  help  the  operator  monitor  the  health  of  the  UAV.  Specifically, 
the  auditory  display  will  present  information  regarding  engine  RPM,  and  the  tactile  display  will  present 
information  regarding  attitude  upset.  We  are  currently  finalizing  the  design  of  the  auditory  and  tactile  display. 
The  use  of  a  multi-modal  display  is  expected  to  improve  SA,  resulting  in  increased  detection  and  faster 
response  times  to  critical  events  during  the  cruise  and  landing  phases  of  a  UAV  operation. 

The  current  project  contrasts  OmniSense  with  a  visual-only  GCS  interface.  The  experimental  task  requires 
participants  to  fly  the  UAV  as  the  primary  task,  while  also  performing  a  secondary  number  monitoring  task 
adapted  from  Sethumadhavan  [21].  The  secondary  task  was  included  to  be  representative  of  a  multi-task 
environment  where  the  participant  needs  to  exhibit  good  performance  in  multiple,  concurrent  tasks.  Operator 
supervisory  control  will  be  assessed  as  a  function  of  display  type,  the  number  of  critical  events,  and  piloting 
experience.  The  project  will  attempt  to  answer  the  following  research  questions: 

a)  Can  a  multi-modal  display  improve  detection  and  response  time  to  critical  events? 

b)  Can  a  multi-modal  display  improve  SA? 

c)  Can  a  multi-modal  display  improve  the  bandwidth  of  information  transfer? 

This  project  will  provide  guidance  on  how  the  output  of  multi-modal  information  can  be  integrated  into  the 
OMI  in  the  IAI  framework.  The  results  of  this  work  will  help  form  the  preliminary  conceptual  framework  to 
design  intelligent  software  agents  that  will  systematically  map  information  to  auditory  and  tactile  displays 


RTO-TR-HFM-170 


5-5 


CAN-3:  SUPERVISORY  CONTROL:  OMNISENSE 


ORGANIZATION 


which  will  serve  as  additional  components  in  the  OMI.  Future  work  will  investigate  the  input  of  multi-modal 
information  and  examine  how  this  can  provide  additional  information  to  the  Operator  State  Assessment. 
This  will  improve  the  accuracy  of  the  Operator  State  Assessment  that  will  be  reported  to  the  Adaptation 
Engine  in  the  IAI  framework  (see  Figure  5-1),  which  in  turn  can  optimize  the  presentation  of  multi-modal 
information  in  the  OMI. 

5.5.3  Stimuli 

The  UAV  simulation  is  developed  in  X-Plane  9.0.  The  simulation  begins  with  the  UAV  set  to  launch  from 
Vancouver  International  Airport,  Canada.  The  conditions  of  flight  are  a  sunny  summer  day  at  noon  in  July. 
The  city  of  Vancouver  is  developed  using  X-Plane’s  software  for  simulating  a  city.  X-plane  has  a  seven  level 
scale  to  determine  the  number  of  objects  in  the  city.  In  our  simulation,  we  used  the  fifth  highest  level  on  the  scale 
for  the  city  of  Vancouver.  However,  no  roadway  vehicles,  or  any  air  traffic  was  simulated.  The  simulated 
environment  for  the  onboard  camera  images  was  generated  using  a  low-fidelity  model  and  X-plane.  Although 
high  fidelity  images  were  not  required  for  this  experiment,  they  can  be  generated  using  Meta-VR  (Brookline, 
MA).  The  simulator  has  been  adapted  to  interface  with  Meta-VR  if  required. 

The  GCS  simulator  has  two  screens,  one  screen  dedicated  to  a  map  display  and  the  Graphical  User  Interface 
(GUI)  used  to  monitor  the  UAV  and  a  second  screen  dedicated  to  the  sensor  view  (e.g.  the  onboard  camera) 
from  the  aircraft.  The  map  display  is  used  for  navigating  the  UAV  and  providing  a  map-based  view  of  its 
location,  as  shown  in  Figure  5-3  (right  screen).  This  consists  of  a  map  displaying  the  city  of  Vancouver. 
An  icon  representing  the  UAV  appears  on  the  map  and  moves  according  to  the  UAV’s  flight  position.  Tasking 
the  UAV  is  initiated  by  having  the  operator  right  click  the  UAV  icon  to  select  commands  from  the  drop  down 
menu  (e.g.  launch  and  land).  Waypoints  are  created  directly  on  the  map  to  navigate  the  UAV  to  fly  specific 
patterns.  To  set  a  waypoint,  the  operator  moves  the  cursor  to  a  position  on  the  map  and  right  clicks  the  mouse. 
A  menu  allows  first  waypoint  and  task  the  UAV  to  fly  to  the  assigned  series  of  waypoints. 


Figure  5-3:  OmniSense  Sensor  Display  and  Map  Display. 

The  GUI  used  to  monitor  the  UAV  is  positioned  to  the  right  side  of  the  map  display  screen.  This  GUI  consists 
of  three  windows: 

a)  A  UAV  status  window; 

b)  A  warning  panel;  and 

c)  An  autoland  panel. 
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The  interface  is  presented  on  the  far  right  side  of  the  window  in  Figure  5-3. 

The  UAV  status  window  provides  information  regarding  the  flight  status,  altitude,  heading,  air  speed,  and  engine 
RPM.  The  altitude  and  air  speed  are  fixed  such  that  the  UAV  cruises  at  approximately  1000  feet/mean  sea  level 
(ft/MSL)  at  100  knots.  This  window  also  has  the  operator  concern  button  that  will  be  used  to  indicate  that  the 
participant  has  detected  a  critical  event. 

The  warning  panel  displays  warnings  and  messages  in  green,  yellow  or  red  depending  on  the  severity  of  the 
warning  or  message.  When  multiple  warnings  are  present,  more  urgent  messages  appear  at  the  top  of  the 
warning  panel,  but  otherwise,  they  appear  in  chronological  order  from  top  to  bottom. 

The  autoland  panel  is  visible  only  when  the  UAV  switches  to  landing  mode  (i.e.,  autoland  mode).  At  the  top 
of  this  panel  is  a  glideslope/localizer  indicator.  This  indicator  uses  a  central  crosshair  to  specify  the  target 
glideslope  and  localizer  point.  An  icon  representing  the  UAV  centres  over  the  crosshair  during  a  trouble-free 
landing,  indicating  that  the  UAV  is  on  the  glideslope  and  localizer  path.  But  if  upon  landing,  the  UAV 
deviates  from  the  glideslope  or  localizer  path,  the  UAV  icon  will  begin  to  deviate  from  the  crosshair, 
providing  the  operator  with  information  on  the  accuracy  of  the  UAV’s  approach.  Immediately  to  the  right  of 
the  glideslope/localizer  indicator  is  an  altitude  indicator  and  below  it,  is  a  lateral  distance  indicator.  The  lateral 
distance  indicator  presents  the  lateral  distance  of  the  UAV  relative  to  the  Touchdown  Point  (TDP).  Both  the 
altitude  indicator  and  the  lateral  distance  indicator  have  the  decision  point  marked  in  red.  The  decision  point  is 
the  point  in  space  in  which  an  abort  landing  can  no  longer  be  performed.  Below  these  indicators  are  several 
numeric-based  indicators  for  lateral  and  vertical  errors,  vertical  descent,  ground  speed,  the  autoland  mode  and 
the  abort  status.  The  abort  button  appears  at  the  bottom  of  this  panel.  If  the  abort  button  is  pressed  before  the 
decision  point  during  a  landing,  the  autoland  will  be  disengaged  and  the  UAV  will  fly  to  a  wave  off  point. 
If  the  UAV  has  passed  the  decision  point,  the  abort  command  will  be  ignored  if  the  abort  button  is  pressed. 

A  second  screen  is  dedicated  to  a  sensor  display  that  provides  a  viewpoint  from  the  rear  right  stabilizer  from 
the  CF  CU-170  Heron  UAV.  With  the  sensor  in  this  position,  the  vantage  point  contains  a  view  of  the  front 
portion  of  the  air  vehicle  (Figure  5-3,  left  screen). 

The  screens  in  Figure  5-3  will  be  divided  into  5  main  Areas  Of  Interest  (AOIs)  for  the  purpose  of  collecting 
eye  movement  data  from  the  participant: 

a)  The  sensor  display; 

b)  The  map  of  Vancouver; 

c)  The  UAV  status  panel; 

d)  The  warning  panel;  and 

e)  The  autoland  panel. 

The  participant’s  eye  gaze  on  each  AOI  will  be  analyzed  for  both  the  baseline  condition  (without  multi-modal 
display)  and  experimental  condition  (with  multi-modal  display). 

5.5.4  Experimental  Design 

The  study  is  a  2  x  2  x  2  mixed  design.  The  OmniSense  display  is  a  between-subject  factor  (visual-only  GCS 
display  vs.  multi-modal  GCS  display).  Flight  experience  (naive  vs.  expert)  is  a  second  between-subject  factor. 
The  naive  group  will  have  no  pilot  experience,  whereas  the  expert  group  will  have  recently  acquired  at  least 
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ten  flying  hours.  The  within-subject  factor  is  the  number  of  critical  events  (no  critical  events  vs.  multiple 
critical  events). 

The  dependent  variables  for  the  UAV  monitoring  task  are  the  number  of  critical  events  detected,  response 
time  to  press  the  operator  concern  button,  response  time  to  press  the  abort  button,  the  participant’s  confidence 
level  in  his/her  monitoring  performance  to  a  critical  event,  perceived  mental  workload  as  measured  via  the 
NASA  Task  Load  Index  (NASA-TLX)  [22],  the  participant’s  SA  measured  using  a  method  derived  from 
Bums  et  al.  [23].  Participant  eye  movements  will  be  monitored  as  a  measure  of  visual  attention.  The  accuracy 
and  the  response  times  for  the  secondary  number  monitoring  task  will  also  be  evaluated  to  assess  the 
participant’s  available  bandwidth  of  information  transfer  when  he/she  is  performing  the  UAV  monitoring  task. 

5.5.5  Apparatus 

The  OmniSense  GCS  simulator  is  based  on  X-Plane  9.0  developed  by  Laminar  Research  (Portland,  OR). 
X-plane  is  a  flight  simulation  environment  that  also  includes  a  plug-in  architecture,  which  allows  users  to 
create  and  modify  their  own  modules.  We  developed  X-plane  to  include  the  Heron,  which  is  a  Medium- 
Altitude,  Long-Endurance  (MALE)  UAV  manufactured  by  Israel  Aerospace  Industries  (Ben-Gurion  Airport, 
Israel).  This  particular  UAV  was  chosen  because  it  is  currently  flown  by  the  CF  in  theatre  in  Afghanistan. 

The  Open  Unmanned  Mission  Interface  (Open  UMI  v.  3.1)  developed  by  Defense  Technologies  Inc.  (Tampa, 
FL)  is  used  to  communicate  between  the  GCS  and  the  X-plane  simulator.  Open  UMI  is  a  common  operator 
control  interface  for  unmanned  systems  that  uses  current  NATO  STANAG  4586  and  Joint  Architecture  for 
Unmanned  System  (JAUS)  standards.  STANAG  4586  requires  a  Vehicle  Specific  Module  (VSM)  to  interface 
between  the  vehicle  protocol  and  STANAG  messages  to  support  the  GUI  for  the  GCS.  The  VSM  and  the  GUI 
for  the  GCS  were  designed  by  InnUVative  Systems,  Inc.  (Ottawa,  ON).  The  OmniSense  GUI  resembles  the 
GUI  used  for  the  United  States  (US)  Army  Shadow  UAV  [24].  The  participant’s  eye  movements  on  the  GCS 
display  (Figure  5-3)  will  be  monitored  using  two  Design  Interactive  flexiGaze  eye  trackers  (Orlando,  FL). 

Customized  software  was  developed  to  run  on  a  separate  computer  for  the  experimenter  to  introduce  system 
faults  (e.g.,  low  engine  RPM  warning,  and  high  engine  RPM  warning)  and  environmental  hazards 
(e.g.,  turbulence,  and  wind  shear)  into  the  UAV  flight.  This  software  allows  the  experimenter  to  pre -program  a 
series  of  faults  and  hazards  or  to  introduce  them  in  real  time  while  the  participant  is  controlling  the  UAV. 
The  experimenter  display  is  presented  in  Figure  5-4. 
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Figure  5-4:  OmniSense  Experimenter  Display. 


5.5.6  UAV  Monitoring  Task 

The  UAV  monitoring  task  is  the  primary  task  in  the  current  study.  The  participant  will  launch  the  UAV, 
command  the  UAV  to  predetermined  waypoints,  and  land  the  UAV.  The  participant  will  also  monitor  for 
potential  critical  events.  If  a  critical  event  occurs,  the  participant  is  instructed  to  respond  by  pressing  the 
appropriate  buttons  ( Operator  Concern  and! or  Abort)  depending  on  the  phase  of  flight. 

Each  flight  scenario  is  divided  into  3  phases: 

a)  Take-off; 

b)  Cruise;  and 

c)  Landing. 

Figure  5-5  shows  each  phase  and  the  key  points  during  each  section  of  the  flight.  Table  5-1  describes  the 
events  during  the  flight  and  the  possible  critical  events  that  may  occur. 
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Figure  5-5:  Events  Associated  with  Each  Phase  of  Flight  (see  Table  5-1  for  event  description). 


Table  5-1 :  Description  of  Each  Event  Associated  with  Each  Phase  of  Flight. 


Phase  of  Flight 

Flight  Position 

Event 

Take-Off 

1 

Launch 

Participant  launches  UAV 

2 

400  ft 

Secondary  task  begins 

3 

700  ft 

Participant  tasks  UAV  to  waypoints 

Cruise 

4 

1st  waypoint 

Possible  critical  event  (e.g.,  high  engine  RPM  or  wind  shear) 

Possible  simulation  pause  (initiate  SA  queries) 

Participant  tasks  UAV  to  recover 

5 

2nd  waypoint 

Possible  critical  event  (e.g.,  high  engine  RPM  or  wind  shear) 

Possible  simulation  pause  (initiate  SA  queries) 

6 

3  rd  waypoint 

Possible  critical  event  (e.g.,  high  engine  RPM  or  wind  shear) 

Possible  simulation  pause  (initiate  SA  queries) 

7 

4th  waypoint 

Possible  critical  event  (e.g.,  high  engine  RPM  or  wind  shear) 

Possible  simulation  pause  (initiate  SA  queries) 

Landing  Approach 

8 

5th  waypoint 

Possible  critical  event  (e.g.,  wind  shear) 

Possible  simulation  pause  (initiate  SA  queries) 

Landing  Touch  Down 
/  Landing  Abort 

9 

Touch  Down  /  Abort 

UAV  lands  on  runway  (secondary  task  ends)  or  landing  is  aborted 

5-10 


RTO-TR-HFM-1 70 


dMNATO 
WP  I  OTAN 


CAN-3:  SUPERVISORY  CONTROL:  OmniSense 


In  the  take-off  phase,  the  participant  launches  the  UAV  from  the  runway.  The  participant  accomplishes  this 
task  by  right  clicking  on  the  UAV  icon  and  selecting  the  launch  command  from  a  drop-down  menu.  This  will 
launch  the  UAV  and  the  participant  will  be  able  to  monitor  its  take-off  from  the  displays.  When  the  UAV 
reaches  400  ft,  the  number  monitoring  task  (described  below)  adapted  from  Sethumadhavan  [21]  begins. 
When  the  UAV  reaches  700  ft.,  the  participant  will  task  the  UAV  to  the  1st  waypoint.  Once  this  command  is 
selected,  the  UAV  will  alter  its  course  and  fly  to  the  1st  waypoint,  entering  the  cruise  portion  of  the  flight. 

While  cruising,  the  UAV  holds  at  approximately  1000  ft  and  flies  through  4  waypoints.  After  crossing  the 
1st  waypoint,  the  participant  will  be  tasked  to  land  the  UAV.  Once  the  land  command  has  been  selected, 
the  autoland  interface  will  appear  on  the  GCS  interface  and  the  UAV  will  lower  its  landing  gear  in  preparation 
to  land. 

When  the  UAV  reaches  the  5th  waypoint,  it  begins  the  landing  portion  of  the  flight.  The  UAV  will  engage  its 
flaps  and  begin  to  descend.  At  this  point,  the  participant  must  watch  the  autoland  panel  and  monitor  the 
landing  of  the  UAV.  When  the  UAV  lands,  it  will  touch  down  at  the  final  point,  which  is  a  runway  at  the  same 
airport  where  the  UAV  took  off.  Once  the  UAV  descends  below  100  ft  /  MSL,  the  secondary  task  ends. 

Critical  events  may  occur  during  the  cruise  and/or  the  landing  phase.  During  the  cruise  phase,  the  participant 
may  encounter  either  system  faults  or  environmental  hazards.  During  the  landing  phase,  the  participant  may 
encounter  an  environmental  hazard.  The  critical  events  will  be  evenly  distributed  across  all  sessions  according 
to  Table  5-2  such  that  each  participant  will  experience  all  combinations  of  system  faults  and  environmental 
hazards.  The  time  of  occurrence  of  each  critical  event  will  be  randomly  determined. 


Table  5-2:  Combinations  of  System  Faults  and  Environmental  Hazards  That  Can  Occur  in  a  Scenario. 


Phase  of  Flight 

Cruise 

Landing 

No  System  Faults  /  No  Environmental  Hazards 

No  Environmental  Hazards 

System  Fault  or  Environmental  Hazard 

No  Environmental  Hazards 

No  System  Faults  /  No  Environmental  Hazards 

Environmental  Hazard 

System  Fault  or  Environmental  Hazard 

Environmental  Hazard 

The  participant  will  be  told  that  the  primary  task  is  to  monitor  and  react  to  critical  events,  while  carrying  out 
the  secondary  task.  If  the  UAV  experiences  a  critical  event  during  the  cruise  phase,  the  participant  will  press 
the  operator  concern  button  immediately  after  detecting  the  critical  event.  If  the  UAV  experiences  a  critical 
event  during  the  landing  phase,  the  participant  will  press  the  operator  concern  button  immediately  after 
detecting  the  critical  event,  and  will  press  the  abort  button  if  the  participant  believes  that  he/she  cannot  land 
the  UAV  safely.  If  the  abort  button  is  pressed  during  a  landing,  the  UAV  will  abort  the  landing  and  fly  to  the 
wave-off  point  that  is  located  at  the  end  of  the  runway. 
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5.5.7  Number  Monitoring  Task 

The  participant  will  perform  a  secondary  task  in  addition  to  monitoring  the  UAV  for  critical  events. 
The  secondary  task  consists  of  monitoring  numbers  adapted  from  Sethumadhavan  [21].  A  series  of  numbers 
between  100  and  199  will  appear  on  the  computer  monitor  at  2-second  intervals.  The  participant  will  be  told 
that  a  number  that  is  less  than  130  or  greater  than  170  represents  a  warning.  The  participant  is  to  press  the 
space  bar  on  the  computer  keyboard  immediately  after  detecting  a  warning. 

5.5.8  Procedure 

Participants  will  be  tested  individually.  The  study  will  be  conducted  over  the  course  of  three  days  that  will  not 
span  more  than  a  week.  The  participant  will  first  receive  training  prior  to  the  experimental  sessions. 
The  training  includes  a  20  minute  multi-media  tutorial  on  some  basic  principles  of  flight  and  procedures  for 
operating  a  UAV.  Following  the  video,  the  participant  will  summarize  the  flight  procedures  to  demonstrate 
that  he/she  understood  the  concepts  in  the  video.  Subsequently,  the  participant  will  be  seated  in  front  of  the 
three  computer  monitors  for  the  duration  of  the  study.  The  experimenter  will  then  calibrate  the  two  eye 
trackers.  The  participant  will  be  familiarized  with  the  UAV  monitoring  task  and  the  number  monitoring  task. 
Subsequently,  the  participant  will  fly  a  practice  scenario  on  the  GCS  simulator. 

Following  familiarization,  the  participant  will  proceed  to  the  experiment.  The  experiment  contains  12  scenarios 
distributed  across  three  sessions.  Session  1  contains  the  previously  mentioned  training  procedure  and  two 
scenarios;  Sessions  2  and  Session  3  each  contain  five  scenarios.  The  order  of  scenarios  will  be  randomized  for 
each  participant  to  control  for  order  effects.  The  duration  of  each  session  is  two  hours;  sessions  will  be  held  on 
separate  days. 

Each  scenario  will  have  3  phases:  take-off,  cruise,  and  landing.  During  each  scenario,  two  SA  queries  will  be 
triggered  at  randomly  predetermined  times,  one  during  the  cruise  phase  and  one  during  the  landing  phase. 
When  triggered,  the  simulation  will  pause  and  the  participant  will  answer  three  questions  chosen  from  the  set 
of  SA  queries.  The  participant  will  also  rate  the  confidence  of  his/her  current  monitoring  performance  on  a 
full  range  confidence  scale.  The  scale  ranges  from  0  -  100%,  where  0%  indicates  that  the  participant 
undoubtedly  has  no  confidence  in  his/her  monitoring  performance  to  a  critical  event  and  100%  indicates  that 
the  participant  is  absolutely  confident  in  his/her  monitoring  performance  to  a  critical  event  [25].  Once  the 
participant  has  answered  these  questions,  he/she  will  click  on  the  resume  button  on  the  screen  and  the 
simulation  will  continue  from  the  point  where  it  paused.  At  the  completion  of  the  scenario,  the  participant  will 
again  rate  the  confidence  of  his/her  monitoring  performance  relative  to  the  entire  scenario  on  a  full  range 
confidence  scale.  The  duration  of  each  scenario  is  approximately  13  minutes,  which  includes  time  for 
answering  the  SA  queries,  and  the  participant  rating  his/her  confidence  in  monitoring  performance  to  a  critical 
event.  Subsequently,  the  participant  will  be  provided  with  a  short  rest  break.  At  the  completion  of  the  last 
scenario  in  the  session,  the  participants’  perceived  mental  workload  will  be  assessed  using  a  computerized 
version  of  the  NAS  A-TLX  [22] . 

5.5.9  Summary 

In  this  Tech  Demo  we  explore  effects  of  multi-modal  display  on  supervisory  control,  SA  of  the  mission 
environment,  and  perceived  mental  workload.  The  effects  of  a  visual  secondary  task  on  operator  performance 
will  also  be  evaluated. 


5-12 


RTO-TR-HFM-1 70 


dMNATO 
WP  I  OTAN 


CAN-3:  SUPERVISORY  CONTROL:  OmniSense 


5.6  UNMANNED  SYSTEMS  USED 

Multi-modal  display  for  simulated  UAV  GCS. 


5.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

As  indicated  in  Table  5-3  the  OmniSense  tech  demo  provides  information  pertaining  to  the  supervisory 
control  technology  design,  and  development.  The  information  was  conveyed  primarily  at  NATO  HFM-170 
meetings.  The  meetings  provided  an  opportunity  to  share  information  on  the  nature  of  supervisory  control 
tasks,  operator  interface  technologies,  and  integration  concepts  that  could  help  enhance  supervisory  control 
performance. 

Table  5-3:  OmniSense  Technology  Demonstration  -  Level  of  Interaction  with  NATO  HFM-170. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

X 

X 

Coordination 

Collaboration 

5.8  SUMMARY  OF  TD  RESULTS 

Empirical  data  collection  has  not  commenced.  The  following  are  preliminary  potential  dependent  variables 
and  associated  hypotheses: 

1)  Critical  event  detection:  participants  will  detect  more  critical  events  and  detect  those  critical  events 
more  quickly  in  the  multi-modal  condition; 

2)  Response  time  to  abort:  the  response  time  to  press  the  abort  button  will  significantly  decrease  in  the 
multi-modal  display  condition; 

3)  Confidence  in  monitoring  performance:  the  confidence  in  monitoring  performance  to  a  critical  event 
will  significantly  increase  in  the  multi-modal  display  condition; 

4)  Dwell  times  on  UAV  monitoring  task:  the  dwell  times  (i.e.,  the  sum  of  consecutive  eye  fixation 
durations  in  a  particular  AOI)  on  the  UAV  monitoring  task  will  significantly  decrease  in  the  multi¬ 
modal  display  condition; 

5)  Secondary  task  accuracy:  accuracy  in  the  secondary  task  will  significantly  improve  in  the  multi¬ 
modal  display  condition; 

6)  Situation  awareness:  the  participant’s  SA  will  significantly  improve  in  the  multi-modal  display 
condition; 

7)  Perceived  mental  workload:  the  participant’s  perceived  mental  workload,  as  measured  by  the  NASA- 
TLX  [22],  will  significantly  decrease  in  the  multi-modal  display  condition;  and 
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8)  Flight  experience:  the  naive  participants  will  show  poorer  performance  than  the  expert  participants  in 
the  baseline  condition,  but  will  not  significantly  differ  in  performance  from  the  expert  group  in  the 
multi-modal  condition.  The  multi-modal  information  is  hypothesized  to  improve  naive  performance 
to  a  greater  extent  than  expert  performance. 


5.9  LESSONS  LEARNED 

The  current  study  is  in  progress.  The  design  and  development  of  the  OMI  component  of  the  IAI  framework 
for  OmniSense  is  nearly  complete.  The  empirical  data  collection  for  the  visual-only  GCS  interface  will  begin 
in  November  2011. 


5.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  experiments  will  be  conducted  in  a  virtual  environment,  not  with  an  actual  UAV. 


5.11  CONCLUSIONS 

The  design  and  development  of  the  OMI  component  of  the  IAI  framework  for  OmniSense  is  nearly  complete. 
The  empirical  data  collection  for  the  visual-only  GCS  interface  will  begin  in  November  2011.  Based  on  earlier 
work  showing  that  multi-modal  displays  enhanced  UAV  monitoring  performance  [13], [14], [15], [16], 
we  anticipate  that  OmniSense  will  enhance  supervisory  control  by  providing  the  human  operators  with  the 
ability  to  perform  real-time  monitoring  of  critical  variables  that  would  otherwise  be  undetected  if  eye  gaze 
was  directed  elsewhere.  The  benefit  of  OmniSense  is  anticipated  to  be  particularly  evident  in  an  increase  in 
the  detection  of  critical  events,  a  reduction  of  response  times  to  critical  events,  and  increased  SA. 
This  suggests  that  the  OmniSense  solution  will  be  more  effective  than  a  visually-only  GCS  interface. 


5.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

The  incorporation  of  a  multi-modal  display  like  OmniSense  into  the  OMI  component  of  the  IAI  framework 
provides  an  example  using  the  intelligent  software  agents  to  interact  with  multi-modal  displays  for  optimizing 
operator-agent  interactions.  Furthermore,  multi-modal  inputs  in  the  form  of  eye  movements  and  speech 
assessment  (e.g.,  loudness,  vocal  emotion)  and  facial  expressions  could  further  enhance  Operator  State 
Assessment.  Future  work  would  support  the  design  and  development  of  other  software  agents  to  manage 
multi-modal  interactions  and  integrate  them  to  other  agents  designed  to  assess  other  operator  states 
(e.g.,  electroencephalography,  and  electrocardiography)  and  environmental  states  (e.g.,  weather,  system  status, 
and  communication  links)  to  enhance  supervisory  control  of  multiple  UAVs. 

The  implication  of  this  study  is  that  multi-modal  displays  linked  with  IAIs  have  the  potential  to  improve 
overall  human-machine  system  performance  if  they  are  designed  properly.  However,  if  designed  improperly, 
IAIs  have  the  potential  to  degrade  system  performance  by: 

a)  Reducing  operator  trust  in  the  automation; 

b)  Presenting  irrelevant  information; 

c)  Presenting  information  that  distracts  the  user;  or  in  the  worst-case  scenario;  and 

d)  Suppressing  information  that  is  currently  required. 
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Additionally,  other  implications  of  this  research  raise  the  issue  of  the  dilemma  for  automation  and  adaptation 
using  IAI  technologies  for  supervisory  control  of  a  UAV. 
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6.1  DATES 

SMAART  (2006  -  2008)  and  SUSIE  (2009  -2011). 

6.2  LOCATION 

Brest  -  Nancy  -  Paris  (France). 


6.3  SCENARIO/TASKS 

The  setting  chosen  for  SMAART  is  the  surveillance  of  a  strategic  air-base,  i.e.,  a  military  air-base  which  can 
deploy  combat  aircrafts  with  nuclear  payloads,  and  is  often  used  for  sensitive  operations  (Figure  6-1). 
Of  course,  such  a  base  has  important  needs  in  the  field  of  security.  In  SMAART,  we  propose  to  introduce 
rotary-wing  UAVs  (among  other  things)  in  order  to  perform  surveillance  tasks  and  to  track  and  identify 
intruders.  The  UAVs  (about  a  dozen)  and  their  collective  decision  algorithms  constitute  the  autonomous 
system  that  the  operator  interacts  with. 
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Figure  6-1 :  Map  of  the  Air  Base  Used  for  Multi-UAV  Surveillance  and  Intrusion  Tracking. 


The  rotary-wing  UAVs  envisioned  in  SMAART  weigh  about  8  kg,  can  travel  at  a  maximum  speed  of 
80  km/h,  have  an  autonomy  of  one  hour  and  are  able  to  detect  intruders  via  optical  sensors  (daylight,  light 
intensification,  infrared).  They  are  able  to  navigate  autonomously  about  the  air  base  at  a  low  altitude,  avoiding 
buildings  and  forbidden  zones,  to  communicate  between  themselves  and  with  the  Ground  Control  Station 
(GCS)  and  their  sensors  allow  them  to  detect  and  eventually  identify  intruders  (Figure  6-2). 


Figure  6-2:  Intrusion  Scenario  Played  on  Simulation. 


6.4  TECHNOLOGIES  EXPLORED 

The  following  paragraphs  describe  the  principles  of  the  self-organizing  multi-agent  system  used  in  our 
demonstrator.  The  aspects  specifically  related  to  the  man-machine  interaction  and  human  factors  will  be 
developed  in  the  next  section. 
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6.4.1  Modes  and  States 

As  with  the  behavior-based  approaches,  the  UAVs  have  a  finite  set  of  base  behaviors  that  they  can  adopt 
during  their  mission.  In  SMAART,  we  refined  behaviors  into  the  two  notions  of  mode  and  state.  At  any  given 
moment,  an  UAV  is  in  a  given  state  that  has  been  determined  by  its  mode.  There  is  often  a  simple  one  to  one 
mapping  between  mode  and  state;  for  example,  an  UAV  in  modePatrol  is  always  in  statePatrol.  But  this  is  not 
always  the  case,  hereafter,  we  list  the  different  modes  and  their  associated  states  between  parentheses  when 
there  is  more  than  one:  modePatrol,  modePursuit,  mode  Auto  (statePatrol,  statePursuit),  modeRally, 
modeHover,  mode  Stop. 


6.4.2  Patrol  Pheromone 

A  stymergic,  virtual  pheromone-based  algorithm  was  developed  to  allow  the  UAVs  (the  agents  of  the  MAS) 
to  coordinate  their  trajectories  in  order  to  patrol  the  air-  base  efficiently  (visit  every  point  as  often  as  possible). 
Stymergy  is  a  method  of  communication  in  emergent  systems  where  the  individual  parts  of  the  system  interact 
with  one  another  by  modifying  their  local  environment.  This  natural  occurrence  has  been  observed  in  ant 
colonies.  Ants  communicate  with  each  other  by  laying  down  pheromones  along  their  trails,  so  where  ants  go 
within  and  around  their  nest  forms  a  stymergic  system. 

Similarly,  the  UAVs  share  a  virtual  grid-like  environment  superimposed  to  the  actual  air  base.  Each  cell  in 
that  grid  stores  a  numerical  values  (quantity  of  patrol  pheromone)  that  is  directly  linked  to  patrol  times: 
the  higher  the  value,  the  more  recently  an  UAV  patrolled  this  cell.  When  a  UAV  enters  a  virtual  cell,  it  adds  a 
fixed  amount  to  the  value  of  the  cell.  As  time  goes  by,  this  pheromone  evaporates  (following  a  cell-based 
evaporation  value),  so  the  longer  a  cell  stays  unvisited,  the  lower  its  pheromone  value  becomes. 

When  an  UAVs  under  statePatrol  has  to  choose  its  movement  (next  cell),  it  chooses  the  nearby  cell  with  the 
lowest  pheromone  value,  i.e.,  the  one  that  was  patrolled  the  longest  time  ago.  This  principle  ensures  that  the 
agents  will  spread  across  the  air-base,  as  they  produce  pheromones  that  repel  each  other. 


6.4.3  Alarm  Pheromone 

In  order  to  pursue  intruders  once  they  are  detected  by  the  system  (by  the  UAVs  or  by  other  means 
e.g.,  perimeter  sensors)  a  pheromone-based  algorithm  has  been  developed  similar  to  the  one  used  for  patrolling. 
The  latter  is  based  on  the  production/avoidance  of  an  evaporating  patrol  pheromone,  while  the  following  pursuit 
algorithm  is  based  on  the  consumption  by  the  UAVs  in  state  statePursuit  of  an  alarm  pheromone  that  is  produced 
each  time  an  intruder  is  detected  and  which  diffuses  in  the  environment  (another  grid). 

Each  time  a  contact  is  detected  a  fixed  amount  of  alarm  pheromone  is  dropped  in  the  corresponding  cell. 
As  time  goes  by,  the  pheromone  from  each  cell  diffuses  in  the  neighboring  cells.  For  a  single  contact,  this  can 
be  viewed  as  the  representation  of  the  evolution  of  the  intruders’  probability  of  presence. 

6.4.4  Alarms  and  Contacts 

SMAART ’s  rotary- wing  UAVs  system  is  not  the  only  security  system  on  the  air-base,  there  are  perimeters 
sensors  on  the  fence,  various  alarm  systems  in  the  buildings,  patrols,  etc.  In  the  SMAART  project,  we  also 
study  the  joint  use  of  fixed-wing  UAVs  and  also  of  a  sensor  network,  but  this  is  outside  the  scope  of  this 
paper.  It  suffices  to  say  that  an  intruder  or  a  group  of  intruders  can  potentially  trigger  a  lot  of  detection 
systems.  For  example,  a  commando  of  three  people  that  breach  the  perimeter  of  the  base  could  be  detected  at 
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the  same  moment  by  the  fence’s  sensors,  one  or  two  rotary-wing  UAVs  and  a  higher  altitude  fixed-wing 
UAV.  This  scenario  could  produce  up  to  3x(l+2+l)  =  12  different  alarms  for  the  same  event,  i.e.,  a  three- 
people  commando  breaching  the  fence. 

In  order  to  prevent  information  overload  for  the  system  as  well  as  for  the  operator,  alarms  that  happen  close  to 
each  other  temporally  and  spatially  are  aggregated  together  in  a  contact.  This  simple  mechanism  depends  on  a 
time  interval  T  (a  few  seconds)  and  a  radius  R  (about  ten  to  twenty  meters).  A  new  contact  is  generated  if  an 
alarm  is  raised  that  is  not  close  enough  (closer  than  R)  to  an  “open”  contact,  i.e.,  a  contact  based  on  an  alarm 
no  older  than  T.  Thus,  the  drops  of  alarm  pheromone  are  generated  upon  detection  of  the  contacts,  not  the 
alarms,  this  prevents  the  formation  of  excessive  spikes  of  pheromone  in  case  of  multiple  detection.  In  a  similar 
way,  the  operator  is  not  presented  the  alarms  themselves,  but  rather  the  contacts  which  are  a  composite  objects 
that  he/she  can  analyze  at  will. 

The  main  results  of  these  algorithmic  approaches  are  displayed  on  the  following  diagrams  (Figure  6-3).  On  the 
left  side,  one  finds  an  example  of  the  initialization  phase,  where  small  circles  represent  the  respective  UAV, 
and  purple  layer  the  level  of  pheromone  (the  more  purple,  he  more  recent  the  area  was  visited).  On  the  right 
side,  the  image  represents  a  stabilized  surveillance  procedure.  One  can  see  that  the  coverage  of  the  area  is 
quite  efficient. 


Figure  6-3:  Main  Results  of  These  Algorithmic  Approaches. 


6.5  HUMAN  FACTORS  ISSUES  EXPLORED 

6.5.1  Framework  Description 

On  the  one  hand,  there  exists  a  very  large  amount  of  literature  in  the  field  of  multi-agent  systems  (MAS, 
a  sub-field  of  Artificial  Intelligence)  devoted  to  enable  a  group  of  artificial  agents  to  accomplish  one  or 
several  tasks  in  cooperation.  On  the  other  hand,  most  of  the  research  on  interaction  between  human  and  semi- 
autonomous  systems  focuses  on  “single  instance”  systems  like  intelligent  cockpits,  industrial  process  control 
system,  etc.  But  there  is  few  work  conducted  on  the  human  control  of  a  multi-agent  system.  The  domain  of 
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multiple  UAVs  control  is  close  to  MAS  control,  see  for  example  the  work  of  Cummings  et  al.  (tactical 
missiles  [1]  or  UCAVs  [2])  or  the  research  around  the  MIIRO  test  bed  (Multi-Modal  Immersive  Intelligent 
Interface  for  Remote  Operation)  [3].  But  these  approaches  do  not  consider  giving  decisional  autonomy  to  the 
agents  (here,  the  UAVs),  the  decision  is  centralized  and  concerns  target  assignments,  individual  path  planning 
and  so  on. 

Controlling  or  supervising  an  actual  multi-agent  system  involves  dealing  with  a  number  of  entities  that  are 
required  to  take  some  level  of  decision  autonomously  and  locally,  i.e.,  using  information  that  may  not  be 
available  to  the  human  controller.  In  the  following  sub-section,  we  review  some  approaches  for  Human  - 
multi-agent  system  control. 

Behaviors  -  The  most  straightforward  way  for  enabling  an  operator  to  control  a  number  of  agents  is  to  endow 
the  agents  with  a  fixed  set  of  basic  behaviors  that  the  agent  is  ale  to  perform  autonomously  (rally  a  point, 
patrol,  follow  a  target,  etc.)  The  task  of  the  operator  is  to  choose  an  appropriate  behavior  for  each  agent  and  to 
monitor  their  progress  in  their  task,  affecting  behaviors  accordingly. 

This  control-by-behavior  paradigm  is  prevalent  in  MAS  control  for  simple  (often  reactive)  agents  whose 
actions  can  be  easily  monitored  -  often  visually.  For  example,  the  RoboFlag  domain  [4]  (a  game  of  capture  - 
the-flag  played  by  two  opposing  agent  teams  under  Human  supervision)  is  particularly  suited  to  this  approach 
[5], [6].  Control-by-behavior  can  be  effective  if  a  small  number  of  behaviors  cover  the  need  of  the  system, 
but  this  approach  loses  its  interest  if  one  needs  more  agents,  more  complex  or  more  numerous  behaviors  as  the 
management  of  individual  agents  becomes  impossible  for  the  operator  [7]. 

Policy  -  In  the  context  of  MAS  control,  the  control-by-policy  approach  would  have  the  advantage  of  sparing 
the  operator  from  the  individual  management  of  agents.  Rather  than  to  assign  individual  goals  or  behaviors  to 
agents,  the  operator  issues  global  constraints  or  advices,  and  the  agents  determine  their  course  of  action 
accordingly.  This  approach  involves: 

•  A  representation  and  expression  system,  usually  close  to  propositional  logic; 

•  An  interface,  usually  text  or  speech-based  (due  to  the  link  between  logic  and  -  constrained  -  natural 
language).  Control-by-policy  is  well  suited  to  mixed-initiative  systems  [8];  and 

•  A  software  architecture  able  to  interpret  policies  and  evaluate  them  against  current  or  hypothetical 
situation  (hence  barring  reactive  agents  in  favor  of  deliberative  ones). 

This  approach  was  used  for  interaction  with  planning  systems  like  SOCAP  (Systems  for  Operations  Crisis 
Action  Planning)  [9], [10]  that  allows  enunciating  constraints  like  “Secure  Air  Superiority  in  Sector  A  before 
Air  Superiority  in  Sector  B”,  “Defend  the  North-East  Sector”  or  “Don’t  employ  more  than  5  sorties  in  Region 
H”.  Other  applications  include  communication  network  management  on  the  battlefield  [11]  or  commercial 
airlines  operations  [12]. 

Playbook  -  The  term  playbook  refers  to  pre-defined  tactics  used  by  football  teams’  coaches.  Rather  than  to 
re-define  from  scratch  and  communicate  to  every  team  member  how  to  behave  for  the  next  play  phase, 
the  coach  refers  to  a  set  of  tactics  known  by  each  team  member  and  only  has  to  instantiate  them  in  the  current 
context  (assign  a  specific  role,  a  variant,  etc.)  This  allows  effective  teamwork  with  few  communications, 
as  each  team  member  knows  each  other’s  role. 

This  is  used  as  a  very  effective  metaphor  for  the  control  of  multi-agent  systems.  The  playbook  becomes  a 
library  of  plans  of  action  that  are  available  for  the  operator  to  instantiate  at  various  levels  of  detail,  hence 
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allowing  various  levels  of  autonomy  for  the  agents.  For  example,  in  a  surveillance  context  the  operator  could 
request  a  reconnoitre  of  an  area  and  leave  the  system  decide  which  agent  to  send  and  which  pattern  to  choose, 
but  he/she  could  also  choose  the  number  of  agents  (even  the  agents  themselves)  or  the  pattern,  or  any 
combination  of  parameters  and  leave  the  system  decide  the  rest. 

The  playbook  approach  was  studied  in  the  context  of  tactical  ground  robots  [13],[14],  real-time  interaction 
with  heterogeneous  military  UAVs  [15].  It  was  also  used  within  RoboFlag  simulations  to  study  variations  of 
operator’s  performance  [16], [17]. 

Proxy  Agents  -  The  purpose  of  a  proxy  agent  is  to  allow  Humans  to  interact  with  a  multi-agent  system. 
Such  an  artificial  -  software  -  agent  is  part  in  a  multi-agent  system  in  which  it  functions  as  either  a  Human’s 
or  an  artificial  agent’s  representative,  i.e.,  communicating,  negotiating  at  his/her  behalf.  A  proxy  agent  can  be 
seen  as  a  common  interface  for  Humans  and  artificial  agents. 

This  approach  considers  the  operator-system  relationship  in  a  “call  center”  perspective  [18],  operators  being 
called  by  the  system  when  it  detects  a  coordination  problem  that  it  cannot  solve.  This  approach  has  the 
advantage  of  blending  together  Humans  and  artificial  agents  with  a  common  “interface”,  but  this  inevitably 
has  some  pitfalls  like  the  lack  of  situation  awareness  of  operators  who  are  called  in  when  the  system  decides 
so,  and  the  agent  team’s  rigid  interaction  strategies. 

SMAART  is  an  exploratory  research  project  funded  by  the  French  Defence  Research  Agency  (Delegation 
Generate  de  l’Armement)  that  aims  at  producing  -  in  simulation  -  the  prototype  of  a  multiple  UAVs  system, 
including  its  control  station.  This  project  motivates  research  in  the  field  of  Artificial  Intelligence/M  AS, 
but  also  in  Human-Information  System  Interaction,  in  this  case  Human-MAS  Interaction.  The  setting  chosen 
for  SMAART  is  the  surveillance  of  a  strategic  airbase,  i.e.,  a  military  air-base  which  can  deploy  combat 
aircrafts  with  nuclear  payloads,  and  is  often  used  for  sensitive  operations.  Of  course,  such  a  base  has 
important  needs  in  the  field  of  security.  In  SMAART,  we  propose  to  introduce  rotary- wing  UAVs  (among 
other  things)  in  order  to  perform  surveillance  tasks  and  to  track  and  identify  intruders.  The  UAVs  (about  a 
dozen)  and  their  collective  decision  algorithms  constitute  the  autonomous  system  that  the  operator  interacts 
with. 


6.5.2  Framework  Applied  to  HFM-170 

The  purpose  of  our  demonstrator  is  three-fold.  In  a  first  step,  we  want  to  demonstrate  that  swarm  intelligence 
is  adapted  to  simple  missions  on  a  dedicated  area,  such  as  surveillance  and  intrusion  tracking.  Secondly, 
we  aim  at  analyzing  the  gap  existing  between  swarm  algorithms  performance  and  limitations  and  the 
perception  operators  may  have  from  these  elements.  Third,  the  demonstrator  (through  its  extension)  proposes 
some  new  interaction  modes  that  can  minimize  operational  semantic  gaps  and  limitations  and  be  more 
intuitive  for  the  users. 

6.5.3  Human  Factors  Issues 

One  can  see  that  the  rotary- wing  UAVs  in  SMAART  can  theoretically  accomplish  their  task  in  a  completely 
autonomous  manner.  That  is,  all  the  UAVs  could  be  set  to  modeAuto,  therefore  patrolling  the  air-base  in  search 
of  intruders  and  switching  to  pursuit  when  they  detect  a  trace  of  alarm  pheromone  which  would  guide  them 
toward  the  intruders.  The  operator’s  only  action  would  be  to  adjust  the  priority  of  some  zones  via  the  pheromone 
evaporation  values:  he/she  would  be  little  more  than  a  spectator.  But  the  dangers  of  full-blown  automation 
without  a  human  in  the  loop  are  well  known,  as  well  as  the  unique  abilities  of  a  human  operator  (pattern 
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recognition,  intuition,  etc.).  Nonetheless,  when  a  Human  is  indeed  in  the  control  loop  with  a  partially 
autonomous  system,  more  often  than  not,  his/her  role  is  to  supervise  or  make  up  for  the  system’s  shortcomings, 
which  leads  to  various  negative  consequences  (on  mental  workload,  situation  awareness,  complacency,  or  skill 
degradation,  see  [19]).  On  the  contrary  we  chose  to  adopt  a  human-  centered  approach  for  the  control  of  the 
UAV  system  in  SMAART.  The  operator  is  at  the  center  of  the  system  and  has  at  his/her  disposal  a  whole  range 
of  interaction  levels  with  the  system.  Depending  of  his/her  workload,  moment,  particular  UAV,  number  and 
localization  of  alarms,  etc.,  the  operator  will  select  the  ones  that  he/she  deems  appropriate  (Figure  6-4). 


Figure  6-4:  Dual  Screen  Man-Machine  Interface  of  SMAART  Simulator. 

He/she  can  for  example  choose  to  let  the  system  operate  autonomously,  assuming  a  supervisory  role,  and  then 
to  begin  to  manage  very  closely  a  few  UAVs  when  an  intrusion  is  first  detected  to  track  it  (leaving  the  other  in 
autonomous  patrol).  Any  combination  is  possible. 

Intruders  are  not  so  common  on  a  strategic  air-base,  therefore  the  main  task  of  the  operator  in  SMAART  is  to 
supervise  the  patrolling  of  the  grounds  by  the  UAVs.  The  UAVs  should  visit  every  accessible  location  in  the 
base  regularly  in  order  to  maximize  the  chance  to  detect  an  intruder.  In  order  to  achieve  this,  the  UAVs  are 
able  to  use  an  algorithm  based  on  a  virtual  patrol  pheromone  that  tends  to  spread  them  evenly  across  the  base. 
They  are  repulsed  by  points  that  have  been  recently  visited  by  an  UAV  and  therefore  seek  less  visited 
locations. 

Despite  the  efficiency  of  this  algorithm,  a  Human  operator  is  in  charge  of  supervising  the  UAVs.  His/her  role 
is  to  make  up  for  the  system’s  eventual  shortcomings,  but  also  to  adjust  the  UAVs’  behavior  in  order  to  take 
into  account  extraneous  constraints  or  various  pieces  of  information  that  cannot  be  easily  translated  into  the 
system’s  representations.  The  interventions  of  the  operator  could  include  for  example:  sending  an  UAV  at  an 
overlooked  location2,  or  that  the  UAVs  concentrate  temporarily  on  a  higher  priority  zone,  making  sure  that 
the  UAVs  avoid  a  certain  area,  etc. 

A  same  hybrid  mode  is  allowed  for  the  tracking  of  intrusion,  where  UAV  can  autonomously  track  alarm 
pheromone  spread  by  the  intruders  or  follow  waypoint  orders  given  by  the  operator  (Figure  6-5). 
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Figure  6-5:  Time-Line  of  Scenarios. 

Tests  were  handled  upon  a  population  of  8  military  subjects  (French  Navy  School  students),  6  men  /  2  women, 
from  20  to  23  years  old.  Quantitative  and  qualitative  data  were  collected  upon  their  performance,  modes  of 
interaction  and  subjective  evaluation  of  system’s  and  own  performance  at  tasks. 

The  main  results  on  the  previous  diagram  (Figure  6-6)  show  that  though  the  operators  usually  believe  the 
system  to  be  far  less  efficient  than  their  own  strategies,  the  average  performance  they  achieve  is  at  most  in  the 
average  performance  bounds,  and  even  slightly  underperforming.  These  results  can  be  interpreted  as  a  clear 
misunderstanding  -  and  mistrust  -  of  the  system  assistance,  leading  to  “interfering”  and  “spoiling” 
intervention  of  the  operators.  This  raises  the  issue  of  man-machine  interface  intelligibility,  in  terms  of 
commands  as  well  as  when  reflecting  the  state  of  the  system. 


Figure  6-6:  Main  Results  for  Patrolling  Phase  -  a)  Average  Idleness 
for  Locations;  b)  Operators  Actions  Review. 

On  the  contrary,  performance  results  on  tracking  and  interception  of  intruders  show  that  operators  are  more 
efficient  (20%  gain)  than  autonomous  tracking,  especially  because  intruders’  “intentions”  are  more  easily 
decoded  by  human-based  situation  assessment  than  simply  following  a  grid-based  digital  map. 

In  order  to  fill  the  gap  between  operators  understanding,  new  research  directions  have  been  defined  that  are 
related  to: 

•  New  means  of  interaction  with  operational  and  tactical  maps  /  UAVs  /  pheromone  maps:  see 
http://recherche.telecom-bretagne.eu/susie/video/. 

•  The  definition  of  elementary  actions  that  could  be  used  for  mapping  pheromone  algorithm  dedicated 
adaptations  to  operational  requirements  that  are  closer  to  operators’  understanding  and  protocols. 
Table  6-1  gives  a  first  list  of  such  elements  (most  relevant  of  them  in  bold  case). 
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Table  6-1:  A  First  List  of  Elementary  Actions  That  Could  be  Used  for  Mapping  Pheromone 
Algorithm  Dedicated  Adaptations  That  Are  Closer  to  Operators’  Understanding 
and  Protocols  (Most  Relevant  of  Them  in  Bold  Case). 


Moving  Target 

Point 

Line 

Contour 

Area 

Monitor 

Insure  a 
permanent 
surveillance  of 
punctual  target 
(restricted 
version  of  area 
monitoring). 

Detect  fixed  or 
moving  target 
present  on  a  line. 
Can  be  extended  to 
the  monitoring  of  a 
graph. 

Detect  fixed  or 
moving  objects 
close  to  a  given 
contour.  Special 
attention  may  be 
put  on  objects 
crossing  the 
contour. 

Detect  fixed  or 
moving  objects 
within  a  given 
area. 

Avoid 

Collision  or  threat 
avoidance.  Implies  the 
perception  of  the 
moving  target  and  a 
minimal  ability  to 
predict  its  trajectory. 

See  avoid  area. 

Avoid  flying  over  a 
line  that  is  known  to 
exist  in  the  area 
(e.g.,  road  or 
highway). 

Avoid  (or  get  the 
UAVs  out  of)  a 
given  area. 

Find 

Find  one  or  several 
moving  targets 
within  a  given  space 
(hypothesis:  objects 
are  present  in  the 
area).  Finding  a 
moving  target  needs 
to  “catch”  it  within 
the  sensor  range  and 
to  detect  it 
successfully. 

Determine  the 
position  of  one 
or  several 
punctual  targets 
known  to  be 
present  on  zone. 

Find  at  least  one 
point  on  a  line  that 
is  known  to  exist  in 
the  environment. 
Possibility  of  taking 
in  account 
complementary 
information  that 
facilitate  the  task 
(road  direction 
constraints,  etc.). 

Close  to  “find  a 
line”  with 
complementary 
notion  of  “inside” 
and  “outside”. 

Same  as  “find  a 
contour”. 

Follow 

Keep  one  (or  several) 
moving  target’s) 
under  detection/ 
tracking  range.  Can 
imply  to  remain 
simultaneously  out  of 
range  for  security  of 
UAV. 

Intercept 

Act  so  as  to  cross  the 
trajectory  of  the 
moving  target  soon  as 
possible.  Needs  to  rely 
on  information  on 
moving  target’s 
trajectory. 

Patrol 

Calibrate  own 
trajectory  on  a  line 
(e.g.,  selected  road). 

Same  as  “patrol  a 
line”. 

Guarantee  a 
“regular”  presence 
within  an  area. 

6.6  UNMANNED  SYSTEMS  USED 

As  described  here  above,  the  Unmanned  System  used  was  not  real  but  only  simulated. 
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6.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

SMAART  software  framework  is  proposed  to  be  shared  amongst  NATO  HFM-170  partners.  The  software 
packages  and  a  basic  user  manual  have  been  communicated  to  the  group  members.  Considering  simple 
adaptation  of  UAV  behaviors,  the  framework  could  help  in  simulating  respective  field  of  study. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

X 

X 

Coordination 

? 

? 

? 

Collaboration 

X 

X 

X 

6.8  SUMMARY  OF  TD  RESULTS 

Main  TD  results: 

•  Proof  of  feasibility  of  surveillance  missions  (area  monitoring  and  intruders  tracking)  using  swarm 
intelligence  controlled  by  human  operators; 

•  Swarms  algorithms  robustness  and  efficiency  checked;  and 

•  Preliminary  design  of  related  man-machine  interfaces. 


6.9  LESSONS  LEARNED 

The  most  important  lesson  learned  is  related  to  the  gap  of  understanding  and  to  the  trust  of  operators  in 
swarms’  algorithm.  Human  factors  studies  have  shown  that  there  is  a  strong  need  of  adapting  commands  and 
system’s  feedback  representations  in  order  to  fill  this  gap  and  to  facilitate  operators  work  while  maintaining 
system  capabilities  (mostly  robustness). 


6.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  study  was  -  and  still  is  -  done  on  simulations  and  not  on  real  UAVs.  Results  have  to  be  confirmed 
statistically  from  an  extended  test  panel. 


6.11  CONCLUSIONS 

Swarm  intelligence  seem  to  be  a  promising  approach  for  multiple  UVs  control  in  terms  of  algorithmic 
performance  and  robustness,  so  far  Human  Factors  and  especially  man-machine  communication  and 
interaction  are  properly  adapted. 


6.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

Future  research  will  focus  upon: 
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•  Swarm  algorithm  adaptation  in  order  to  enlarge  supported  functions  to  a  broader  spectrum  of 
operational  missions. 

•  Semiotic  engineering  of  man-machine  interface  in  order  to  adapt  displays  and  commands. 

•  New  modalities  of  man-machine  interface  in  order  to  support  the  meaningfulness  of  interaction 
(see  perspectives  in  [20]  ,[21],  [22]) . 
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7.1  DATES 

PEA  FHPA:  2007-2011. 

7.2  LOCATION 

Paris,  France. 

Demonstration  done  at  LTO  -  Laboratoire  Technico-Operationnel  DGA  (Arcueil  /  France). 


7.3  SCENARIO/TASKS 

The  second  French  Tech  Demo  is  related  to  an  upstream  program  named  “Human  Factors  and  Human/ 
Automate  Authority  Sharing  in  Unmanned  Aerial  Systems”.  The  main  objective  of  this  program  is  to  define 
new  means  of  cooperation  and  interaction  between  Humans  and  Automates,  based  on  the  concept  of 
“Authority  Sharing”.  In  practical  terms,  it  is  intended  to  optimize  the  workload  of  existing  UAV  systems  by 
allocating  dynamically  the  operators’  functions,  allowing  thus  the  integration  of  multiple  UAVs  and  payloads 
without  necessarily  augmenting  the  number  of  operators. 

The  program  is  organized  in  4  phases  during  36  months: 

•  Phase  1 :  “RETEX”  (RETour  d’EXperience)  -  experience  feedback  from  the  French  Army; 

•  Phase  2:  Search  for  innovative  solutions  on  HF  and  Authority  Sharing; 

•  Phase  3:  Implementation  of  the  innovative  solutions;  and 

•  Phase  4:  Experiments  /  HF  evaluation. 

The  scenario  envisioned  for  this  project  sets  two  Ground  Control  Stations  (GCS)  collaborating  together 
toward  the  identification  of  a  common  enemy: 

•  The  first  GCS  controls  two  tactical  UAVs  (fixed-wing  UAVs),  with  one  payload  each,  flying  at  two 
different  locations  on  the  map  (in  the  same  geographical  area). 
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ORGANIZATION 


•  The  other  GCS  controls  one  MALE  UAV  (fixed-wing),  with  one  payload,  flying  in  the  same 
geographical  area  as  the  others. 


Figure  7-1:  Tactical  AVO’s  Cartography  Screen. 
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Figure  7-2:  Tactical  AVO’s  Manual  Control  Screen. 
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ORGANIZATION 


Figure  7-3:  Tactical  PO’s  Camera  Views  Screen. 


Each  station  welcomes  a  2-member  team:  an  Aerial  Vehicle  Operator  (AVO)  and  a  Payload  Operator  (PO). 
At  the  beginning  of  the  scenario,  each  team  is  unaware  of  the  other  team’s  presence  in  the  area. 

The  tactical  GCS’s  mission  is  two-fold:  UAVl’s  goal  is  to  open  a  road  for  a  convoy  by  detecting  and 
identifying  all  the  potential  targets  along  that  road  while  UAV2  is  watching  the  convoy  and  its  surroundings. 
The  MALE  GCS’s  mission  is  to  watch  the  activity  along  a  border. 

At  a  certain  point  of  the  scenario,  the  MALE  UAV  is  rerouted  to  a  meeting  area  but  an  air  traffic  lane  appears 
and  prevents  it  from  reaching  the  meeting  point  on  time.  One  of  the  tactical  UAVs  is  then  rerouted  to  the 
meeting  area  and  shares  its  payload  (EO  camera)  with  the  MALE  GCS.  The  video  feedback  provided  allows 
the  MALE  station  to  perform  its  mission  and,  as  the  air  traffic  lane  closes,  the  MALE  UAV  can  be  directly 
directed  to  the  meeting  point. 


7.4  TECHNOLOGIES  EXPLORED 

Within  the  scenario  presented  before,  two  different  concepts  are  assessed:  an  “authority  sharing”  concept, 
between  the  AVO  and  the  automate  controlling  each  UAV  (throughout  the  mission),  and  a  human-human 
collaboration  concept  between  the  two  payload  operators  (while  sharing  the  video  feed). 
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7.4.1  Authority  Sharing 

After  the  definition  of  the  three  different  operative  modes  (automatic,  intermediary  and  manual),  each 
corresponding  to  different  levels  of  function  allocation,  a  set  of  HMI  was  designed  to  support  each  operative 
mode  with  a  focus  on  the  trajectory  management  macro-task. 


Figure  7-4:  Examples  of  Ul  Designed  to  Support  the  Different  Operative  Modes: 
(a)  Draggable  Vector  Tool  and  (b)  Manual  Controls. 


7.4.2  Human-Human  Collaboration 

The  human-human  collaboration  part  of  this  project  focuses  on  the  payload  sharing  between  the  two  ground 
control  stations.  During  the  mission,  the  tactical  payload  operator  “lends”  the  payload  of  one  UAV  to  the 
MALE  payload  operator.  Two  levels  of  sharing  are  defined:  one  where  the  full  control  of  the  tactical  payload 
is  transferred  to  the  MALE  operator,  and  the  second  where  only  the  video  feedback  is  sent  to  the  MALE 
operator  while  the  control  remains  under  the  tactical  operator’s  responsibility. 
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ORGANIZATION 


Figure  7-5:  View  of  the  MALE  PO  Screen  When  Receiving  a  Shared  Camera. 

The  HMI  designed  to  support  these  2  concepts  have  been  tested  on  a  simulation  environment  during  two 
experimentation  campaigns  with  UAV  military  personnel  from  the  French  Army  and  the  French  Air  Force. 


7.5  HUMAN  FACTORS  ISSUES  EXPLORED 

7.5.1  Reference  Frame 

One  of  the  project’s  phases  purpose  was  to  design  an  “authority  sharing”  engine  which  function  is  to 
dynamically  allocate  the  functions  to  either  the  automata  or  the  operator,  depending  on  the  operational 
context.  This  “operational  context”  is  mainly  defined  by  different  criteria: 

•  Number  of  UAVs; 

•  UAVs’  objectives  and  status; 

•  Types  of  missions  /  tactical  environment;  and 

•  Meteorology  (specially  gusts  of  wind). 

Refining  the  classical  approach  of  autonomy  levels  [4], [6],  we  defined  a  methodology  based  on  Proud’s  Level 
Of  Autonomy  (LOA)  matrix  [5]  and  Boyd’s  OODA  loop  [1]  in  order  to  derive  a  general  framework  where 
different  levels  of  function  allocation  can  be  coupled  with  the  4  phases  of  Boyd’s  loop  (Observe,  Orient, 
Decide,  Act).  Three  different  operative  modes  (automatic,  intermediary  and  manual)  were  implemented  in  the 
system,  each  corresponding  to  different  levels  of  function  allocation  [2], [7],  thus  covering  the  spectrum  of 
autonomy  configurations  [3]. 
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7.5.2  Human  Factors 

During  our  experimentation  campaigns,  we  assessed  the  impact  of  our  new  designs  and  the  underlying 
concepts  on  operators’  performance  and  workload. 

Operators  played  the  scenarios  several  times  with  the  “authority  sharing”  engine  activated  or  not. 
This  allowed  us  to  observe  and  evaluate  the  effects  of  letting  the  computer  decide  to  whom  (between  the  AVO 
and  the  automation)  each  task  is  allocated  throughout  the  scenario. 

The  same  factors  (performance  and  workload)  were  assessed  with  the  payload  operators  during  the  camera 
sharing  phase:  we  studied  how  well  the  MALE  operators  performed  their  enemy-seeking  task  in  two 
configurations:  when  fully  controlling  the  tactical  UAV’s  camera  or  only  viewing  the  video  feedback 
(and  thus  giving  instructions  to  the  tactical  PO). 


7.6  UNMANNED  SYSTEMS  USED 

As  described  above,  the  Unmanned  Systems  used  were  not  real  but  only  existing  in  a  simulation  environment. 
It  can  be  noted  though  that  the  GCS  environment  was  reproduced:  each  team  (1  aerial  vehicle  operator  and 
1  payload  operator)  was  alone  in  a  shelter-like  room.  The  Mission  Planner  was  remotely  giving  audio 
instructions  to  the  teams. 


7.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

When  the  NATO  HFM-170  Meeting  took  place  in  Paris  (September  2009),  only  a  small  part  of  the  program 
has  been  communicated  to  its  members.  Indeed,  the  Phase  4  (experiments  /  HF  evaluation)  of  the  program 
hadn’t  started  yet. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

Coordination 

Collaboration 

7.8  SUMMARY  OF  TD  RESULTS 

1)  A  methodology  based  on  Proud’s  Level  Of  Autonomy  (LOA)  matrix  and  Boyd’s  OODA  loop  has  been 
used  and  tested,  in  order  to  derive  a  general  framework  where  different  levels  of  function  allocation  can  be 
coupled  with  the  4  phases  of  Boyd’s  loop  (Observe,  Orient,  Decide,  Act).  Three  operative  modes  corresponding 
to  different  levels  of  function  allocation  were  then  defined,  implemented,  tested  and  validated. 

2)  The  Authority  Sharing  Tool  does  not  vary  significantly  the  overall  performance  whether  it  is  activated  or 
not,  but  the  test  panel  size  didn’t  allow  us  to  statistically  confirm  this  data.  However,  to  illustrate  this  result, 
the  table  below  shows  the  performance  measured  during  a  communication  breakdown  (workload  increased). 
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Overall  Performance 

Workload  Level 

Average 

Standard  Deviation 

Average 

Standard  Deviation 

AS  tool  active 

65,50 

8,52 

3,00 

1,15 

AS  tool  inactive 

77,28 

13,94 

2,50 

1,29 

3)  AVOs  may  control  two  UAVs  at  the  same  time  when  (a)  the  workload  level  is  acceptable  and  (b)  they  are 
assisted  by  an  “authority  sharing”  tool,  but  only  if  (i)  the  operators  trust  the  tool’s  choices  and  (ii)  the  choices 
help  ensuring  UAV’s  safety. 

4)  POs  are  not  able  to  perform  two  missions  with  two  different  payloads,  although  operators  may  increase 
their  situation  awareness  level  if  these  two  payloads  are  used  for  one  mission  and  target  the  same  area. 
Regarding  the  transfer  modes,  POs  always  preferred  to  keep  control  over  the  payload  while  performing  the 
target-seeking  task. 


7.9  LESSONS  LEARNED 

The  most  important  lesson  learned  is  related  to  the  relationship  that  humans  have  with  automata  (the  “authority 
sharing”  engine)  capable  of  allocating  in  real  time  their  tasks,  sometimes  distributing  them  the  machine.  Indeed, 
its  acceptance  degree  is  directly  related  to  the  situation  awareness  held  by  the  operators  and  their  trust  in  the 
automation. 


7.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  study  was  carried  out  on  a  simulation  environment  and  not  with  real  UAVs.  However,  the 
experimentation  campaigns  involved  several  UAV-related  military  personnel.  Results  have  to  be  confirmed 
statistically  from  an  extended  test  panel. 


7.11  CONCLUSIONS 

The  operators  appreciated  the  HMIs  designed  during  the  program,  in  particular  the  “draggable  vector  tool” 
(it  allows  the  operator  to  easily  reroute  the  UAV  in  a  drag-and-drop  motion).  Regarding  the  “authority 
sharing”  engine,  the  overall  performance  does  not  change  with  or  without  the  activation  of  the  engine,  but  the 
test  panel  was  too  small  to  statistically  confirm  this  data. 


7.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

Future  research  in  the  area  should  emphasize  the  following  points: 

•  Managing  the  transitions  between  operating  modes  and  related  man-machine  configurations  so  that 
the  operators  would  not  be  handling  too  important  gaps  in  subsequent  configurations  of  the  system;  and 

•  Extend  the  human  factors  analysis,  both  in  quantitative  and  qualitative  way,  through  respectively  an 
extended  panel  that  will  guarantee  a  better  statistical  reliability,  and  a  focus  on  the  instrumentation  of 
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operators’  states  (stress,  fatigue,  focus  of  attention)  so  that  the  understanding  and  tuning  of  the 
different  operating  modes  could  be  more  adequate. 
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8.3  SCENARIO/TASKS 

In  future  military  helicopter  missions  a  critical  function  is  to  get  real-time  surveillance  and  reconnaissance 
from  several  locations  or  targets  at  the  same  time  without  exposing  humans  to  possible  threats.  Therefore, 
the  deployment  of  multiple  UAVs  as  remote  sensor  platforms  in  a  Manned-Unmanned  Teaming  (MUM-T) 
scenario  is  investigated.  The  guidance  of  these  will  be  realised  by  the  commander  aboard  the  mission  leading 
helicopter,  so  the  UAVs  may  be  deployed  flexibly  and  directly  from  where  the  surveillance  and 
reconnaissance  results  are  needed.  This  incorporates  an  operator  to  vehicle  ratio  smaller  than  one.  Since  the 
commander  already  works  on  many  other  tasks  like  mission  management,  system  management, 
communication  or  supporting  the  pilot,  the  addition  of  the  UAV  guidance  and  mission  management  tasks  will 
be  very  demanding  for  him/her,  especially  in  situations  when  the  environment  requires  the  mission  to  be 
re-planned.  On  the  other  hand,  when  missions  last  a  long  time  without  need  for  action,  the  operator  can 
become  inattentive.  These  conditions  may  result  in  reduced  performance  or  even  accidents.  Studies  of 
accidents  with  ground-based  UAV  guidance  attest  that  causes  are  not  only  technical  malfunctions  but  also 
human  error  [1].  Furthermore,  UAV  guidance  experiments  at  the  Institute  of  Flight  Systems  show  that 
operators  produce  errors  which  reduce  mission  performance  [2].  These  errors  result  from  typical  reasons  like 
unbalanced  workload  conditions,  interface  handling  problems,  reduced  situation  awareness,  degraded  operator 
attention,  vigilance  decrements  or  complacency. 

Therefore,  the  main  objective  is  to  reduce  the  workload  of  the  helicopter  commander  to  ensure  mission 
success.  The  approach  includes  shifting  UAV  guidance  from  the  typical  waypoint-based  level  to  a  more 
abstract  task-based  level  to  reduce  the  workload  of  the  operator  and  avoid  overtaxing  due  to  the  multitude  of 
various  detailed  system  management  and  scheduling  tasks  [3], [4].  This  is  realised  by  an  artificial  cognition- 
based  agent  aboard  each  UAV,  which  understands  the  operator-given  tasks  and  generates  tactical  sense 
making  behaviours.  Therefore,  the  commander  provides  single  or  a  series  of  high-level  tasks  to  each  UAV  via 
a  graphical  user  interface  based  on  a  moving-map  display.  Ref.  [5]  describes  this  concept  and  the 
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experimental  findings  in  some  more  detail.  Furthermore,  an  adapted  crew  coordination  concept  is  defined, 
which  shifts  some  system  management  and  communication  tasks  to  the  pilot  flying.  An  assistant  system  for 
the  pilot  flying  helps  him/her  to  adopt  the  new  tasks  and  also  takes  over  the  support  of  the  pilot  flying,  which 
further  reduces  the  workload  of  the  commander.  Finally,  an  assistant  system  for  the  commander  shall  be 
developed,  which  helps  him/her  in  unbalanced  workload  conditions,  improves  situation  awareness  and 
attention  [2], [6], [7]  and  eventually  improves  mission  performance.  The  concept  and  evaluation  of  the 
commander  assistant  system  will  be  presented  in  this  article. 


8.4  TECHNOLOGIES  EXPLORED 

Research  on  pilot  assistant  systems  has  been  conducted  at  the  Institute  of  Flight  Systems  since  the  early 
1990s.  Several  prototypes  (cf.  CASSY,  e.g.,  [8];  CAMA,  e.g.,  [9])  have  also  been  successfully  tested  in  real 
flight.  From  this  experience,  Onken  and  Schulte  describe  the  general  approach  to  assistant  systems  in  a 
broader  context  [10],  which  may  in  turn  be  applied  to  the  multi-UAV  guidance  domain. 

8.4.1  Work  Process  Analysis-Based  Requirements  for  Assistant  Systems 

The  general  approach  by  Onken  and  Schulte  describes  assistant  systems  from  a  more  abstract,  work  process 
oriented  point  of  view.  Figure  8-1  shows  the  structure  of  the  physical  entities  performing  a  work  process  in  a 
work  system. 


knowledge  of 
work  objective 
necessary 


pursues  the  work  objective 


knowledge 
about  human 
operator  & 
cooperation 
necessary 


cooperates  with  human  operator  &  coordinates  tasks 


environment 

knowledge 

necessary 


knowledge  of 
OSM  status  & 
capabilities 
necessary 


takes  environmental  conditions  into  consideration 


makes  use  of  operation-supporting  means 


Figure  8-1:  The  Assistant  System  as  Part  of  the  Work  System. 

The  work  system  is  defined  by  the  work  objective  (arrow  from  the  left),  which  should  be  accomplished  by  the 
work  process,  thereby,  providing  a  result  (arrow  to  the  right).  The  work  system  itself  consists  of  the  Operating 
Force  (OF,  left  in  figure),  which  always  incorporates  a  human.  In  Figure  8-1  the  OF  is  extended  by  the 
assistant  system  (robot  head).  To  fulfil  the  work  objective  the  OF  applies  Operation  Supporting  Means 
(OSM,  right  in  figure),  e.g.,  automation  or  in  our  case  UAVs  and  a  manned  helicopter.  Constraining  factors  to 
the  work  process  are  the  environmental  conditions  (arrow  from  top).  On  this  level  the  interaction  between  the 
OF  and  the  OSMs  can  be  described  with  the  supervisory  control  paradigm  [11].  Onken  and  Schulte  [10] 
characterise  some  properties  of  the  assistant  system  resulting  from  its  integration  into  the  work  system, 


8-2 


RTO-TR-HFM-1 70 


dMNATO 
WP  I  OTAN 


GER-1:  COGNITIVE  AND  COOPERATIVE  ASSISTANT 
SYSTEM  FOR  AERIAL  MANNED-UNMANNED  TEAMING  MISSIONS 


e.g.,  the  assistant  system  shall  also  pursue  the  work  objective.  These  properties  were  refined  to  form  a  set  of 
four  properties  as  depicted  below  each  work  system  in  Figure  8-1.  To  fulfil  them,  the  assistant  system  needs  to 
have  knowledge  in  four  different  areas  (bold  arrows),  i.e.,  the  work  objective ,  the  environment , 
the  cooperation  with  the  human  operator  and  the  OSM.  Knowledge  about  the  OSM  can  be  further  divided  into 
knowledge  about  the  current  state  and  about  how  to  apply  them.  Also,  operator  knowledge  is  split  up  into 
knowledge  about  the  current  operator  state  and  about  interaction  with  the  human  operator. 


8.5  HUMAN  FACTORS  EXPLORED 

Onken  and  Schulte  [10]  also  characterise  the  mentioned  cooperation  between  human  operator  and  assistant 
system  by  the  postulating  basic  behaviour  requirements  for  assistant  systems: 

“Requirement  1: 

The  assistant  system  has  to  be  able  to  present  the  full  picture  of  the  work  situation  from  its  own 
perspective  and  has  to  do  its  best  by  own  initiatives  to  ensure  that  the  attention  of  the  assisted  human 
operator(s)  is  placed  with  priority  on  the  objectively  most  urgent  task  or  sub-task. 

Requirement  2: 

If  according  to  requirement  1  the  assistant  system  can  securely  identify  as  part  of  the  situation 
interpretation  that  the  human  operator(s)  cannot  carry  out  the  objectively  most  urgent  task  because  of 
overtaxing,  then  the  assistant  system  has  to  do  its  best  by  own  initiatives  to  automatically  transfer  this 
situation  into  another  one  which  can  be  handled  normally  by  the  assisted  human  operator(s). 

Requirement  3: 

If  there  are  cognitive  tasks,  the  human  operator(s)  is(are)  principally  not  capable  to  accomplish,  or  which 
are  of  too  high  risk  or  likely  a  cause  of  too  high  costs,  these  tasks  are  to  be  allocated  to  the  assistant 
system  or  operation  supporting  means,  possibly  a  supporting  cognitive  unit.” 

Based  on  these  requirements  a  more  detailed  concept  for  the  assistant  system’s  intervention  and  cooperation 
with  the  operator  will  be  derived  in  the  next  section.  Afterwards,  the  corresponding  types  of  intervention  for 
the  triggers  will  be  defined,  which  are  also  derived  by  use  of  the  requirements  stated  above. 

8.5.1  Intervention  Triggers 

Referring  to  the  introduction,  an  assistant  system  intervention  should  be  triggered  to  prevent  human  overload 
and  eventually  human  error,  which  may  lead  to  reduced  performance  or  accidents.  Therefore,  the 
identification  of  error  causes  is  the  trigger  for  an  assistant  system  intervention.  Several  models  for  human 
error  causes  and  prevention  have  already  been  stated,  e.g.,  by  Reason  [12], [13]  or  Hollnagel  [14], [15]. 
The  basic  requirements  stated  above  also  describe  error  causes  and  means  to  prevent  these  from  a  point  of 
view  which  is  very  close  to  a  system  design.  Therefore,  the  basic  requirements  were  used  to  derive  the 
following  three  intervention  triggers: 

1)  Attention  Deficit 

If  according  to  the  first  requirement  an  assistant  system  should  ensure  attention  to  the  most  urgent 
task  it  has  to  intervene,  if  the  operator  does  not  pay  attention  to  the  most  urgent  task.  This  intervention 
is  primarily  meant  to  support  the  operator’s  situation  awareness  [16]. 

2)  Overtaxing 

According  to  the  second  requirement  an  assistant  system  should  intervene,  if  the  operator  is  overtaxed 
in  carrying  out  the  most  urgent  task  to  balance  the  workload.  From  our  point  of  view,  overtaxing  can 
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occur  in  every  step  of  the  human  information  process,  which  roughly  consists  of  information 
acquisition  and  analysis,  decision  selection  and  action  implementation  (cf.  Parasuraman,  Sheridan  and 
Wickens  [17]).  Since  overtaxing  as  part  of  the  information  acquisition  and  analysis  is  already  covered 
by  the  first  intervention  trigger,  only  overtaxing  in  the  decision  selection  and  action  implementation 
process  is  used  as  an  intervention  trigger. 

3)  Likely  Inacceptable  Costs 

Finally,  according  to  the  third  requirement,  an  assistant  system  should  become  active,  if  the  execution 
of  a  task  by  the  operator  would  produce  inacceptable  costs.  If  he/she  is  not  capable  of  accomplishing 
the  task,  this  also  means  inacceptable  task  costs.  This  trigger  is  especially  hard  to  detect  because  the 
system  on  the  one  hand  has  to  decide  upon  the  capabilities  of  the  operator  and  on  the  other  hand  has 
to  make  a  prediction  about  the  future  evolvement  of  the  situation. 

8.5.2  Intervention  Types 

After  identifying  the  triggers  when  an  assistant  system  should  intervene,  knowledge  about  how  it  should  act  is 
necessary.  Intervention  types  basically  represent  different  levels  of  automation  and  interaction.  Here,  several 
studies  and  theories  have  been  published.  Rouse  and  Rouse  distinguish  three  levels  of  automation, 
i.e.,  “manual”,  “management-by-consent”  and  “management-by-exception”  [17].  Another  theory  is  Sheridan’s 
ten  levels  of  automation  in  decision  and  action  support  [18].  Endsley  [19]  first  defined  five  levels  of 
automation  ranging  from  manual  control  to  full  automation  and  later  refined  them  to  ten  levels  of  automation 
[20].  Onken  and  Schulte  [10]  also  distinguish  three  different  styles  of  cooperative  assistance,  i.e.,: 

•  Alerting  -  The  assistant  detects  inadequate  human  behaviour  and  draws  the  attention  of  the  human 
towards  the  corresponding  task,  if  the  human  does  not  have  situation  awareness  about  this  fact.  It  may 
give  advice. 

•  Associative  -  The  assistant  continuously  presents  proposals  but  does  not  actively  draw  the  attention 
of  the  operator.  The  operator  can  task  the  assistant  to  automatically  carry  out  the  corresponding  task. 

•  Substituting  -  The  assistant  can  either  temporarily  or  permanently  take  over  commitments  of  the 
human.  Temporarily  substituting  assistance  may  be  authorised  by  the  human  or  intervene  without 
waiting  to  be  authorised.  Permanently  substituting  assistance  can  be  authorised  in  the  beginning  or 
built  in  by  design. 

Considering  an  attention  deficit  as  an  intervention  trigger,  assistance  in  situation  awareness  is  needed.  Here, 
alerting  assistance  was  selected  as  an  intervention  type  to  interact  with  the  operator.  By  guiding  the  attention, 
situation  awareness  about  the  most  urgent  task  is  transmitted  to  the  operator.  Inadequate  behaviour  not  only 
incorporates  the  detection  of  necessary  action  in  a  certain  task  but  also  the  conclusion  that  the  action  is  urgent 
and  more  important  than  other  tasks.  Additionally  to  alerting  assistance  the  system  shall  not  only  draw  the 
attention  towards  the  task  but  also  improve  situation  awareness  by  telling  the  operator  about  the  reason  for  the 
need  for  action.  Advices  in  terms  of  decision  aids  should  not  be  given  directly  but  on  request.  When 
overtaxing  is  the  reason  for  an  intervention,  the  workload  of  the  operator  needs  to  be  reduced  by  simplifying 
the  task.  Here,  the  intervention  type  associative  assistance  has  been  selected.  The  presentation  of  proposals 
shall  reduce  overload  in  decision  selection,  but  in  contrast  to  associative  assistance  these  should  not  be  given 
continuously  but  only  in  case  of  overtaxing.  The  automated  execution  of  the  task  on  request  is  also  suitable  for 
reducing  overload  in  action  implementation.  Additionally  to  associative  assistance,  the  assistant  system  should 
configure  the  user-interface  for  execution  of  the  task  to  reduce  overtaxing  in  action  implementation  through 
interface  handling  problems.  Inacceptable  task  costs  can  only  be  avoided  if  the  assistant  carries  out  the  task 
itself  or  passes  it  to  an  OSM.  This  intervention  type  is  close  to  temporarily  substituting  assistance,  which 
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intervenes  without  waiting  to  be  authorised.  Here,  cooperation  is  still  necessary  in  terms  of  letting  the  operator 
know,  what  the  assistant  system  did.  If  this  is  not  the  case,  situation  awareness  problems  on  the  part  of  the 
operator  could  occur.  Summing  up,  three  different  intervention  types  were  defined  corresponding  to  the  three 
intervention  triggers:  attention  guidance ,  task  simplification  and  task  reallocation. 

8.5.3  Prioritisation 

The  defined  triggers  and  corresponding  types  of  intervention  now  form  three  different  assistant  functions 
derived  from  the  three  basic  requirements.  Since  there  can  be  several  assistance  needs  at  a  time,  the  additional 
workload  posed  on  the  human  operator  by  several  interventions  at  once  may  overtax  him/her.  Therefore, 
the  operator  is  supposed  to  handle  only  one  intervention  concurrently,  which  means  that  the  different 
functions  must  be  prioritized. 

First  a  prioritisation  is  defined  in  case  the  same  function  is  triggered  multiple  times  at  once.  If  the  operator’s 
attention  should  be  guided  towards  several  urgent  tasks,  not  only  the  urgency  but  also  the  importance  of  tasks 
is  considered  by  the  assistant.  Within  the  same  task  (e.g.,  the  operator  has  to  define  the  next  mission  task  for 
UAV1  and  UAV2  concurrently)  the  urgency  is  taken  as  a  basis  for  deciding  upon  the  higher  priority. 
In  contrast  to  this,  between  tasks  of  different  types  the  task  the  output  of  which  can  have/has  an  effect  on  the 
other  task’s  input  is  prioritized  higher.  Tasks  with  a  lower  priority  will  be  processed  afterwards.  If  the  operator 
is  working  on  several  (not  urgent)  tasks  in  parallel  and  is  overtaxed,  only  the  task  which  the  operator  is  really 
currently  working  on  shall  be  assisted.  Task  reallocation  however  can  always  be  executed  directly,  since  the 
operator  is  not  actively  involved.  Yet,  the  information  dialog  about  the  intervention  needs  to  be  queued,  if  the 
assistant  already  conducts  other  dialogs  with  the  operator. 

The  following  prioritisation  was  defined  between  the  different  assistant  functions: 

1 )  T ask  reallocation; 

2)  Attention  guidance;  and 

3)  Task  simplification. 

Since  task  reallocation  can  and  also  needs  to  be  executed  directly  because  of  the  criticality  for  the  mission  and 
for  safety,  this  function  has  the  highest  priority.  Finally,  the  second  requirement  states  that  the  assistant  system 
shall  only  trigger  a  task  simplification,  if  the  operator  already  works  on  the  most  urgent  task.  Therefore, 
attention  guidance  to  the  most  urgent  task  has  a  higher  priority  than  task  simplification. 

8.5.4  Operator  Interactions 

Only  the  assistant  system  takes  the  initiative  for  an  intervention  and  starts  dialogs  with  the  operator.  However, 
the  operator  has  different  possibilities  to  react  upon  the  interventions  by  interacting  either  with  the  operation 
supporting  means  or  the  assistant  system.  In  case  of  an  attention  guidance  the  operator  has  the  following  three 
reaction  possibilities: 

1)  The  operator  accepts  the  advice  and  switches  to  the  respective  task.  The  assistant  system  recognizes 
the  respective  task  as  the  current  operator  task  and  the  dialog  disappears.  In  the  end  the  task  should  be 
accomplished.  Otherwise  the  advice  reappears. 

2)  The  operator  requests  support  in  the  task  from  the  assistant.  The  assistant  system  switches  to  the  task 
simplification  intervention  type  and  offers  decision  and  action  support. 
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3)  The  operator  does  not  accept  the  advice  and  ignores  the  dialog.  Here,  he/she  needs  to  tell  the  assistant 
to  ignore  the  need  for  action,  which  is  also  remembered  by  the  assistant. 

In  case  of  a  task  simplification  intervention  the  operator  has  similar  options: 

1)  The  operator  accomplishes  the  task  and  the  dialog  disappears.  Here,  it  is  not  relevant  whether  the 
proposed  solution  or  a  different  one  is  executed.  Only  the  result  that  the  task  is  accomplished  is 
important. 

2)  The  operator  requests  automated  execution  of  the  task  from  the  assistant.  The  assistant  system 
switches  to  task  reallocation  intervention  type. 

3)  The  operator  does  not  want  to  work  on  the  task  anymore  and  ignores  the  dialog.  As  soon  as  the 
operator  switches  to  a  different  task,  the  dialog  disappears.  Again  the  reaction  is  remembered  by  the 
assistant. 

In  case  of  a  task  reallocation  intervention  no  further  interaction  of  the  operator  is  possible.  The  assistant  only 
sends  a  dialog  to  the  operator  telling  which  actions  were  performed.  However,  the  operator  can  interact  with 
the  assistant  in  advance  to  prohibit  task  reallocation  for  certain  tasks. 

8.5.5  Specific  Assistant  Knowledge 

According  to  the  defined  concept  Figure  8-2  shows  the  specific  knowledge  needed  by  an  assistant  system  and 
relates  it  to  the  knowledge  areas. 
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Figure  8-2:  Specific  Knowledge  of  the  Assistant  System  as  Part  of  the  Work  System. 
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First  of  all  to  ensure  operator  attention  the  assistant  needs  to  know  the  currently  most  urgent  task. 
For  knowledge  of  the  most  urgent  task  it  has  to  know  which  tasks  have  a  need  for  action.  After  that  it  can 
determine  the  urgent  matters  and  the  most  urgent  one.  For  this  it  is  also  necessary  to  know,  which  tasks  have 
to  be  generally  worked  on  as  part  of  the  work  process.  This  knowledge  can  be  derived  from  the  work  objective 
and  the  available  OSMs.  Since  the  system  shall  only  assist  tasks,  which  are  assigned  to  the  operator  and  not  to 
the  automation,  knowledge  is  required  about  the  current  task  assignment  between  human  and  automation. 
After  recognizing  the  most  urgent  task,  the  assistant  needs  to  know,  if  the  operator’s  attention  is  placed  on  this 
task,  that  means  it  needs  to  know  the  actual  operator  task.  For  balancing  the  workload,  it  is  essential  to  detect 
overtaxing  of  the  operator.  To  be  able  to  offer  appropriate  help,  it  also  needs  knowledge  about  the  actual 
operator  task,  which  causes  the  overtaxing.  Finally,  to  keep  the  costs  on  a  moderate  level,  the  assistant  system 
should  recognize  if  there  are  tasks  which  likely  produce  inacceptable  costs ,  if  they  stay  assigned  to  the 
operator.  Of  course  this  knowledge  is  always  connected  with  a  distinct  uncertainty  since  on  the  one  hand  the 
system  predicts  operator  behaviour  and  on  the  other  hand  it  defines  an  arbitrary  cost  limit  above  which  it 
intervenes.  If  the  assistant  system  decides  that  an  intervention  is  necessary,  it  has  to  know  how  to  conduct  a 
dialog  with  the  operator  and  how  to  give  hints  on  the  user-interface  to  present  the  situation.  Moreover  it  needs 
domain-specific  knowledge  about  which  dialogs  and  advices  are  useful  for  the  operator.  Finally,  to  reallocate 
a  task  the  system  needs  to  know  which  commands  have  to  be  sent  to  the  OSMs  to  accomplish  tasks. 

The  derived  knowledge  shown  in  Figure  8-2  is  still  quite  unstructured.  Therefore,  the  next  section  will 
introduce  a  knowledge-based  implementation  technology  also  proposed  in  [10],  which  allows  a  more 
structured  modelling  of  the  knowledge  which  is  also  closer  to  an  implementation. 

8.5.6  Cognitive  Modelling 

The  assistant  system  for  the  UAV  operator  is  implemented  as  an  Artificial  Cognitive  Unit  (ACU)  to  realize 
goal-driven  behaviour.  For  this,  the  “Cognitive  Process”  (CP)  by  Putzer  was  used,  which  implements  the 
knowledge-based  level  of  the  human  performance  model  stated  by  Rasmussen  [21].  The  CP  has  been 
developed  as  a  model  of  human  information  processing,  which  is  suitable  for  generating  human  like  rational 
behaviour  [22]  in  technical  systems.  The  behaviour  is  completely  defined  by  explicitly  represented 
knowledge,  which  is  split  up  into  goals  the  ACU  shall  fulfil  [1],  action  alternatives  it  can  choose  from, 
schedules  to  implement  them  and  environment  models  to  build  up  a  situational  understanding.  The  CP  was 
implemented  with  the  Cognitive  System  Architecture  (COS A),  described  in  detail  in  [10]. 

In  this  section  the  specified  knowledge  for  an  assistant  system  is  structured  according  to  the  cognitive  process 
(cf.  Figure  8-3).  The  aim  is  to  derive  specific  knowledge  classes  for  a  cognitive  implementation  of  an 
application  independent  knowledge  package  “Assistance”. 
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Figure  8-3:  Application-Independent  Knowledge  Package  “Assistance”. 


The  application  of  the  different  knowledge  classes  to  an  assistant  system  should  start  with  a  definition  of  the 
DESIRES  to  be  achieved  [22].  A  desire  is  a  state  of  the  work  process  that  shall  be  achieved.  If  the  state  is 
different,  the  desire  becomes  an  active  goal.  In  our  case  the  intervention  triggers  describe  states,  where  the  state 
is  not  satisfying  for  the  assistant.  Therefore,  the  goal  of  the  assistant  is  to  achieve  the  opposite  state, 
i.e.,  the  operator  works  on  the  most  urgent  task ,  the  operator's  workload  is  balanced  and  the  costs  are 
acceptable.  The  next  step  in  deriving  knowledge  classes  is  to  figure  out,  which  abstract  ACTION 
ALTERNATIVES  could  be  applied  to  fulfil  the  desires.  These  alternatives  correspond  to  the  intervention  types 
specified  earlier.  Therefore,  if  the  operator  is  not  working  on  the  most  urgent  task,  the  assistant  system  should 
direct  the  attention  of  the  operator.  In  case  the  operator  workload  is  unbalanced,  the  assistant  system  simplifies 
the  task  by  offering  partial  automation  to  the  operator.  If  the  task  costs  would  become  inacceptable,  the  assistant 
shall  reallocate  the  task.  To  execute  the  action  alternatives,  INSTRUCTION  MODELS  are  used,  which  describe 
the  technical  output  of  the  system.  According  to  the  knowledge  areas  illustrated  above,  the  output  can  be  either 
to  the  operator  or  the  OSM.  Outputs  to  the  operator  may  be  to  start  a  dialog  by  sending  a  message  or  adapting 
the  user-interface  to  give  additional  hints  or  to  configure  the  display.  Outputs  to  the  OSM  would  result  in 
sending  instructions  to  the  OSM  to  execute  tasks.  To  understand  the  situation,  be  able  to  check,  if  the  above 
mentioned  desires  are  met  and  also  to  send  appropriate  messages  and  instructions,  an  assistant  system  needs 
ENVIRONMENT  MODELS.  In  this  case  they  correspond  to  the  specific  knowledge  stated  above  (e.g.,  work 
objective,  tasks,  task  assignment,  actual  operator  task,  need  for  action).  For  a  complete  knowledge-based 
modelling  also  rules  are  necessary,  which  instantiate  these  knowledge  classes  and  change  their  attributes. 
The  desire  “operator  works  in  most  urgent  task”  is  instantiated,  if  there  is  an  instance  of  a  “most  urgent  task”  but 
no  instance  of  an  “actual  operator  task”  which  links  to  the  same  task.  The  action  alternatives  are  instantiated 
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according  to  the  prioritisation  defined  above  and  activate  the  corresponding  instruction  models.  The  environment 
models  in  this  package  are  actually  abstract  classes,  which  are  inherited  by  domain-specific  classes  in  a  separate 
knowledge  package  “Mission”.  For  example  in  a  troop  transport  mission  there  is  an  instantiation  of  a  “work 
objective”  class,  which  contains  that  the  troops  have  to  be  at  a  certain  position  at  the  end  of  the  mission. 
The  “Mission”  package  also  contains  instruction  models,  e.g.,  a  message  for  guiding  the  attention  of  the  operator 
towards  the  identification  of  a  certain  object  or  commands  to  the  UAVs  for  adding  or  deleting  mission  tasks. 

8.5.7  MUM-T  Specific  Assistant  Functions 

The  assistant  functions  are  defined  on  the  one  hand  by  the  different  intervention  types  in  the  application- 
independent  knowledge  package  and  on  the  other  hand  by  the  application-specific  tasks  in  the  “Mission” 
package,  which  shall  be  supported.  Therefore,  a  task  analysis  for  the  commander  in  a  manned-unmanned 
teaming  domain  was  performed.  Tasks  in  the  area  of  communication  and  system  management  were  not 
considered  because  in  the  crew  coordination  concept  these  tasks  were  intentionally  shifted  to  the  pilot  flying 
by  experimental  design.  Results  are  shown  in  Figure  8-4. 
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Figure  8-4:  Task  Analysis  for  the  Helicopter  Commander  in  a  Manned-Unmanned  Teaming  Domain. 


On  top  level  tasks  can  be  divided  into  mission  planning  and  mission  execution.  Mission  planning  consists  of 
defining  mission  tasks  (e.g.  departure,  transit,  route/area  reconnaissance  or  surveillance),  assigning  them  to 
either  a  manned  helicopter  or  a  UAV  and  sorting  them  into  the  agent’s  agenda.  To  execute  mission  tasks, 
they  have  to  be  activated  first.  Most  of  the  time  activation  is  done  in  the  order  given  by  the  agenda.  Mission 
task  execution  for  the  manned  helicopters  mainly  consists  of  flight  management  tasks  like  route  planning, 
waypoint  activation  and  waypoint  tracking.  Since  the  UAVs  have  cameras  as  a  payload,  ground  mapping, 
object  recognition,  tracking  and  identification  also  needs  to  be  done  for  reconnaissance  tasks.  Since  the 
helicopter  is  controlled  by  the  pilot  and  the  UAVs  are  guided  on  a  task-based  level  [3],  only  the  highlighted 
tasks  have  to  be  performed  by  the  commander/UAV-operator.  These  tasks  are  also  supported  by  the  assistant 
system.  According  to  the  intervention  types  (attention  guidance,  task  simplification,  and  task  reallocation) 
each  task  can  be  assisted  in  three  different  ways.  The  only  exception  is  the  object  identification  task,  where  a 
task  reallocation  in  terms  of  an  automated  identification  seems  hardly  feasible  with  the  current  state  of  the  art. 


8.6  UNMANNED  SYSTEM  USED 

The  UAVs  used  in  the  simulation  are  generic  vehicles  with  a  flight  performance  close  to  the  manned  platform, 
so  they  are  able  to  fly  in  a  common  mission  in  loose  formation. 
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INTERACTIONS 

The  MUM-T  focus  is  on  specific  requirements  of  the  German  Armed  Forces.  Still,  it  is  of  great  interest  to 
discuss  the  system  concept,  experimental  design  and  results  with  NATO  partners. 
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8.8  SUMMARY  OF  TD  RESULTS 

In  May  2011,  during  an  experimental  campaign  for  the  MUM-T  research  at  the  UBM  the  benefit  and 
cooperation  aspects  of  the  described  assistant  system  were  evaluated.  One  of  the  experimental  missions  used 
in  this  campaign  was  also  shown  as  a  tech  demo  for  the  NATO-HFM  group  in  September  2010. 

8.8.1  Experimental  Setup 

The  primary  objective  for  the  test  persons  was  to  successfully  complete  a  helicopter  troop  transport  mission 
into  hostile  territory.  To  achieve  this,  they  had  additional  three  UAVs,  which  were  guided  on  a  task-based 
level.  The  UAVs  implemented  capabilities  like  route/area  reconnaissance  and  surveillance  in  order  to  secure 
the  routes/landing  sites/objective  for  the  helicopter  and  the  troops.  Each  mission  started  at  a  main  operation 
base  in  friendly  territory.  Here,  the  commander  was  mainly  busy  entering  a  mission  plan  for  the  helicopter  and 
the  UAVs.  After  activating  the  take-off  and  passing  the  corridor  into  insecure  territory  the  UAVs  started  with 
the  reconnaissance  and  the  commander  had  to  observe  the  progress  and  identify  objects  preselected  by  the 
UAVs.  During  the  mission  also  three  events  for  a  major  mission  re -planning  occurred:  identification  of  hostile 
troops  at  the  primary  drop  zone,  detection  of  a  SAM-site  at  the  primary  egress  corridor  and  a  follow-up  troop 
transport  mission  after  completion  of  the  first  troop  transport.  The  experimental  run  was  finally  stopped  after 
30  to  45  minutes  when  the  helicopter  returned  to  friendly  territory. 

Each  test  person  had  to  complete  one  mission  with  and  one  mission  without  support  by  the  assistant  system. 
To  prevent  expectations  about  the  mission  development,  missions  differed  concerning  the  mission  area  and 
the  threat  configurations. 

Eight  German  Army  helicopter  pilots  participated  as  test  persons  at  an  average  age  of  37  years  (min  28  years, 
max  51  years.  Their  flying  experience  ranged  from  830h  up  to  5100h  with  an  average  of  1815h.  The  test 
persons  were  grouped  into  fixed  crews  consisting  of  two  members  (alternating  in  the  roles  of  commander  and 
pilot  flying).  Each  test  person  had  two  days  of  training  for  the  commander’s  workstation.  The  training  began 
with  an  instructed  phase  and  continued  with  a  free  training  phase.  Furthermore  subjects  had  the  possibility  to 
observe  other  subjects  in  their  training  (passive  training). 
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The  experiments  took  place  in  the  helicopter  simulator  of  the  UBM,  which  was  refined  to  support  a  manned- 
unmanned-teaming  mission  including  UAV-guidance  from  the  commander’s  seat.  This  workstation  provides 
two  displays  with  various  formats  especially  for  UAV  tasking  and  object  identification  as  well  as  one  control 
and  display  unit  (in  this  case)  especially  for  entering  mission  goals  and  constraints  and  activating  automated 
helicopter  mission  planning.  The  commander  was  equipped  with  a  headset  for  communication  within  the 
cockpit,  with  other  external  entities  (e.g.,  tower,  AW  ACS)  and  with  the  assistant  system. 

During  each  mission,  the  simulation  was  paused  three  times  in  order  to  measure  the  crew’s  workload  (with 
NASA  TLX  [23])  and  the  situation  awareness  (with  SAGAT  [24]).  The  first  pause  was  made  while  the 
helicopter  was  still  in  friendly  territory,  the  second  in  hostile  territory  outside  the  operation  area  (only  NASA- 
TLX)  and  the  third  inside  the  operation  area.  Video  and  audio  recordings  were  taken  and  all  relevant 
simulation  data  was  logged,  which  included  the  system  interactions  of  the  commander  and  the  pilot  flying. 
After  the  experimental  runs  the  test  persons  were  interviewed  about  acceptance  of  the  assistant  functions  and 
possible  impacts  on  situation  awareness  and  workload. 

The  measured  data  shall  prove  that  the  employment  of  the  assistant  system  can  increase  mission  performance, 
reduce  workload  and  increase  situation  awareness  of  the  commander.  Also  the  interventions  should  be 
considered  reasonable  and  be  well  accepted  by  the  test  persons. 

8.8.2  Experimental  Results 

First  of  all,  mission  performance,  which  is  dependent  from  both  the  commander,  the  pilot  and  from  the 
corresponding  assistance  configuration,  is  presented.  Afterwards,  the  individual  situation  awareness  and 
subjective  workload  are  examined.  Finally,  the  objective  individual  behavior  of  the  commander  and  subjective 
ratings  of  the  assistant  system  functions  by  the  test  persons  are  presented. 

Performance  -  Mission  planning  for  the  UAVs  and  the  helicopter  as  well  as  punctual  activation  of  UAV 
mission  tasks  had  an  effect  on  performance.  If  the  commander  did  not  recce  the  helicopter  route  with  the 
UAVs  in  time,  the  helicopter  either  had  to  wait  until  the  route  was  recced  or  fly  across  insecure  territory. 
To  measure  the  performance  impacts  in  case  the  helicopter  waited,  mission  delays  were  assessed  by 
generating  a  standard  solution  for  each  mission  offline  and  comparing  it  to  the  measured  mission  durations. 
Then  the  frequency  of  mission  delay  in  segments  of  2.5  minutes  each  was  counted. 

Here,  in  the  unassisted  configuration,  excessive  mission  delays,  which  exceeded  7.5  minutes,  occurred  in  four 
cases,  whilst  this  could  be  observed  only  once  in  the  assisted  configuration  (cf.  Table  8-1).  Furthermore, 
the  duration  was  measured  in  which  the  helicopter  had  a  geographical  position  (2D)  that  had  not  been 
photographed  by  a  UAV  before.  In  the  configuration  without  assistant  system,  this  exposure  time  was  three 
times  higher  (t(  1 4)  =  1.74,  p  =  0.1). 
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Table  8-1 :  Comparison  of  Mission  Delay  for  the  Helicopter. 


Delay 

Unassisted 

Assisted 

d  <  2.5  min. 

3 

3 

2.5  min.  <  d  <  5.0  min. 

1 

2 

5.0  min.  <  d  <  7.5  min. 

0 

2 

7.5  min.  <  d 

4 

1 

Overall 

8 

8 

Finally,  performance  of  the  commander  in  the  object  identification  task  was  measured.  Throughout  all  sixteen 
experimental  missions  113  objects  were  located  by  the  UAVs’  ATR  functionality  and  had  to  be  identified 
consequently  (53  without  /  60  with  commander  assistant  system).  Identification  correctness  was  equally  good 
(only  one  error  in  each  configuration)  but  not  analyzed  because  it  was  not  supported  by  the  assistant  system. 
Instead  a  closer  look  was  taken  into  critical  events,  where  the  helicopter  came  close  to  the  unidentified  object 
(15  without  /  10  with  assistant  system)  and  fast  task  completion  times  were  important.  In  the  assisted 
configuration  the  assistant  system  guided  the  attention  of  the  commander  towards  the  object  identification  task 
and  additionally  offered  task  simplification  in  terms  of  display  configuration.  Attention  guidance  reduced 
mean  duration  from  ATR  recognition  until  the  commander  started  the  object  identification  from  46.8  seconds 
unassisted  to  18.2  seconds  assisted,  which  shows  weak  significance  (t(23)  =  1.79,  p  =  0.087).  The  automated 
display  configuration,  which  was  used  five  times  reduced  the  mean  time  for  object  identification  itself  from 
17.3  seconds  to  7.5  seconds,  but  no  significant  effects  could  be  verified  (t(21)  =  0.771,  p  =  0.45).  The  overall 
time  from  ATR  recognition  to  task  completion  of  the  identification  was  reduced  from  66.1  seconds  unassisted 
to  27.7  seconds  assisted,  which  also  shows  weak  significance  (t(21)  =  1.96,  p  =  0.063). 

Situation  Awareness  -  Situation  awareness  was  measured  objectively  using  the  SAGAT  [24]  method.  During 
the  simulation  pauses,  subjects  had  to  estimate  positions  of  own  and  hostile  forces  in  an  electronic  map 
display.  Civil  forces  were  not  counted,  since  subjects  very  often  considered  them  not  relevant  and  omitted 
them  intentionally.  The  estimated  positions  were  compared  to  the  true  positions  at  this  time.  Each  object  was 
awarded  with  two  points  if  distance  between  specified  and  actual  position  was  less  than  0.75  nm,  with  one 
point  if  distance  was  less  than  1.5  nm,  and  no  points  if  distance  was  larger  or  the  object  was  missing.  In  the 
unassisted  configuration  108  out  of  154  points  were  gained  (70.1%),  while  in  the  assisted  configuration  105 
were  gained  (68.2%),  which  shows  nearly  no  difference. 

Moreover,  subjects  were  interviewed  about  the  current  situation  (threats,  reconnaissance  status,  next 
communication  task,  current  and  subsequent  tasks  of  UAVs  and  helicopter).  Correct  and  nearly  correct 
answers  were  awarded  with  two  points,  still  acceptable  answers  with  one  point,  and  wrong  or  missing  answers 
with  no  points.  Again,  for  every  test,  the  points  were  summed  up  and  divided  by  the  maximum  number  of 
points  to  receive  a  percentage.  In  the  interview  the  subjects  gained  significantly  better  results  in  the  assisted 
(95.4%)  than  in  the  unassisted  configuration  (90.0%)  (t(14)  =  2.49,  p  =  0.026).  It  is  also  worth  mentioning  that 
the  test  results  were  showing  very  high  absolute  values  in  both  configurations. 

Subjective  Workload  -  Subjective  workload  was  measured  using  the  NASA-TLX  [23]  subjective  workload 
assessment  tool,  which  divides  workload  into  the  following  six  categories:  Mental  Demand  (MD),  Physical 
Demand  (PD),  Temporal  Demand  (TD),  Performance  (P),  Effort  (E)  and  Frustration  (F). 
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Workload  across  all  phases  and  subjects  shows  a  non-significant  decrease  in  mean  from  42.3%  (unassisted)  to 
38.8%  (assisted)  (t(43)  =  0.714,  p  =  0.48).  The  NASA-TLX  scores  did  not  produce  significant  results,  because 
the  individual  utilization  of  the  scale  varies  more  than  the  differences  between  the  system  configurations. 
However,  the  main  difference  between  these  two  values  was  caused  by  a  significant  decrease  in  temporal 
demand  from  13.7%  (unassisted)  to  8.3%  (assisted)  (t(43)  =  2.28,  p  =  0.027).  Changes  in  other  workload 
dimensions  range  only  from  0.3  -  1.6  %  and  are  not  significant. 

Behavior  -  The  commander’s  interaction  with  the  assistant  system  was  evaluated  by  video  analysis  of 
the  assistant  system  interventions  (system-  or  user-initiated)  and  the  corresponding  commander’s  reactions. 
User-initiated  attention  guidance  was  not  possible  by  design.  Also,  not  in  every  task  knowledge  for  system- 
initiated  task  simplification  or  task  reallocation  was  implemented  (cf.  Table  8-2). 

Table  8-2:  Assistant  Interventions  and  Commander’s  Reactions. 


Object 

Identification 

Single 

Mission  Task 
Planning 
(UAV) 

Mission 

Task 

Activation 

(UAV) 

Mission 

Planning 

Overall 

System 

User 

s 

u 

s 

U 

s 

u 

S 

U 

Overall 

Attention  Guidance 

12 

20 

15 

27 

74 

74 

Accepted 

12 

11 

11 

21 

55 

55 

Not  Accepted 

0 

9 

4 

6 

19 

19 

Task  Simplification 

5 

12 

6 

0 

4 

2 

12 

17 

29 

Accepted 

5 

8 

3 

4 

2 

8 

14 

22 

Not  Accepted 

0 

4 

3 

0 

0 

4 

3 

7 

Task  Reallocation 

8 

5 

4 

2 

5 

14 

19 

91 

31 

122 

Across  all  experimental  runs  74  times  the  assistant  system  intervened  with  attention  guidance.  55  times  the 
commander  complied  with  the  advice  by  switching  to  the  respective  task.  In  the  remaining  cases  subjects  did 
not  work  on  the  respective  task  directly  after  the  advice.  Thus  according  to  the  test  persons,  in  about  75%  of 
the  cases  attention  guidance  was  correct.  Task  simplification  was  evaluated  by  analyzing,  if  the  subjects  used 
the  decision  selection  and  action  implementation  support  offered  by  the  assistant.  Since  the  assistant  could  not 
offer  a  solution  for  object  identification,  here,  only  the  configuration  of  the  user- interface  to  prevent  interface¬ 
handling  problems  was  evaluated.  22  out  of  29  times  (about  75%)  task  simplification  was  used  by  the 
subjects.  Only  single  mission  task  planning  proposals  were  not  accepted  several  times,  but  still  more  than  60% 
(11  out  of  18)  of  the  proposals  were  accepted  either  by  requesting  automated  task  reallocation  or  manually 
inserting  the  mission  task  into  the  UAV’s  agenda.  System- initiated  task  reallocation  was  only  implemented  for 
a  safety  critical  situation,  when  a  UAV  came  close  to  a  threat  and  therefore  was  stopped  by  the  assistant.  Here, 
the  commander  could  not  react,  but  the  intervention  could  be  regarded  as  correct,  since  no  UAVs  came  under 
fire  in  the  assisted  configuration  compared  to  five  times  without  the  assistant.  Also,  in  the  interviews  all  test 
persons  stated  that  the  intervention  was  necessary. 

Ratings  -  In  the  following,  the  most  important  results  of  the  questionnaires,  which  were  given  to  the  test 
persons  during  the  corresponding  mission  debriefings,  are  presented  (cf.  Figure  8-5). 
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Figure  8-5:  Rating  of  Different  Intervention  Types  Across  the  Corresponding  Commander  Tasks. 


Most  of  the  test  persons  accepted  the  attention  guidance  for  object  identification  and  mission  re -planning  as 
necessary  and  stated  that  it  improved  their  situation  awareness.  Attention  guidance  for  activation  and  planning 
of  single  UAV-tasks  produced  more  or  less  indifferent  results.  This  was  because  test  persons  sometimes 
wanted  to  employ  the  UAVs  differently  from  the  assistant  which  led  to  the  understanding  that  the  attention 
guidance  was  too  early  and  not  necessary.  According  to  the  subjects’  statements,  the  task  simplification  in 
terms  of  automated  display  configuration  for  the  object  identification  task  reduced  workload.  Task 
simplification  for  activation  and  planning  of  single  UAV-tasks  were  also  found  to  reduce  operator  workload 
slightly.  For  task  simplification  in  mission  re-planning,  which  was  not  used  very  frequently,  the  subjects 
attested  a  strong  relief  in  workload.  Task  reallocation  was  also  not  used  by  all  test  persons,  but  was  rated  to 
increase  efficiency. 


8.9  LESSONS  LEARNED 

An  interesting  finding  was  that  test  persons  followed  different  strategies  in  mission  planning.  For  example 
some  test  persons  wanted  the  UAVs  to  secure  the  operation  area  and  wait  there  until  the  helicopter  arrive, 
while  others  sent  them  directly  back  to  recce  the  route  to  the  main  operation  base.  Also  some  test  persons 
recced  alternate  routes  whilst  others  focused  on  reconnaissance  of  the  main  route.  This  made  it  difficult  for  the 
assistant  to  propose  mission  plans  which  were  in  accordance  with  the  individual  test  person’s  view.  So,  test 
person’s  strategies  are  highly  individual,  which  might  be  the  consequence  of  too  little  training  on  the 
particular  job  or  too  little  developed  procedures,  since  multi-UAV  guidance  from  the  cockpit  is  a  novel  task  to 
German  Army  helicopter  pilots. 

Another  surprising  point  was  that  although  tasks  were  strictly  divided  upon  commander  and  pilot  flying  in 
terms  of  a  crew  coordination  concept,  task  assignments  were  shifted.  The  commander  even  acted  as  an 
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assistant  for  the  pilot  flying  upon  pilot  errors  although  this  should  not  have  been  necessary  because  the  pilot 
was  also  supported  by  another  assistant  system.  Since  the  commander  assistant  system  did  not  have 
knowledge  about  these  tasks  it  did  not  have  the  full  situational  picture  and  in  one  case  triggered  attention 
guidance  to  a  currently  less  urgent  object  identification  task. 


8.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  results  are  based  on  a  simulated  environment  and  not  a  real-world  campaign.  Therefore  some  aspects  like 
e.g.,  data  link  losses  were  not  regarded.  Since  there  was  only  the  possibility  to  simulate  one  manned  aircraft, 
only  intercom  between  pilot  flying  and  commander  and  radio  communication  to  airport  towers  or  the  mission 
leader  was  simulated.  Radio  communication  to  other  aircraft  was  missing.  This  caused  reduced  workload  for 
the  helicopter  crew  /  UAV-operator. 


8.11  CONCLUSIONS 

A  generic  approach  for  the  development  of  a  knowledge-based  assistant  system  was  presented.  The  approach 
was  adapted  to  the  domain  of  manned-unmanned-teaming,  i.e.,  guidance  of  multiple  UAVs  from  the 
commander’s  workplace  in  a  helicopter  cockpit  aided  by  an  assistant  system.  The  approach  was  evaluated  by 
conducting  experiments  in  the  helicopter  simulator  of  the  UBM.  The  introduction  of  the  assistant  system 
improves  human  factors  related  variables  like  situation  awareness  and  workload,  improves  performance  and 
safety  and  is  well  accepted. 


8.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

The  next  steps  to  further  improve  the  assistant  system  performance  and  acceptance  should  be  to  refine  the 
knowledge  models  for  operator  overtaxing  estimation,  current  task  recognition  and  cost  prediction.  In  addition 
the  action  and  decision  support  for  tasks,  which  include  several  steps  should  be  refined.  Finally,  the  cooperation 
and  variable  task  assignment  between  commander  and  pilot  flying  have  to  be  investigated  closer  and  be 
regarded  within  the  concept. 
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9.1  DATES 

June  2010. 


9.2  LOCATION 

The  experiments/demonstrations  were  held  at  Fort  Benning  GA,  USA. 


9.3  SCENARIO/TASKS 

For  soldiers,  visual  information  is  crucial  in  building  up  situation  awareness.  Not  surprisingly,  when  robots  are 
used  for  reconnaissance  of  a  remote  area,  visual  sensors  are  most  important.  Most  robots  are  therefore  equipped 
with  visual  sensors,  but  not  with  any  other  kind  of  sensor!  This  is  surprising,  because  an  approaching  car 
(yet  invisible  because  still  around  the  comer),  someone  moving  in  an  adjacent  room,  a  slamming  door,  or  the 
loading  of  a  gun,  are  important  events  during  reconnaissance  which  cues  are  primarily  auditory.  When  such 
sounds  occur,  human  beings  almost  instinctively  direct  their  heads  (eyes)  to  the  sound  source  for  visual 
inspection  before  deciding  to  hide,  or  to  make  contact,  to  get  out  very  fast,  to  attack,  etc.  Even  more  so,  human 
beings  immediately  know  where  to  hide  or  where  the  safe  exit  is  because  of  their  excellent  spatial  situation 
awareness  that  results  from  the  human-intrinsic  integrated  perception  of  visual,  auditory,  and  proprioceptive 
information.  We  hypothesize  that  if  such  intrinsic  integrated  multi-modal  perception  would  be  facilitated  in 
remote  perception  using  robots  (by  having  headtracking  control  for  robot’s  sensor  system  that  includes  stereo 
vision  and  spatial  3D  audio,  a  setup  we  refer  to  as  Telepresence  [1]),  spatial  situation  awareness  would  boost 
performance  in  a  robot  reconnaissance  mission.  This  hypothesis  was  investigated  in  the  experiment  reported 
here,  which  was  conducted  as  part  of  a  research  collaboration  between  the  US  Army  Research  Lab,  ft  Benning, 
and  TNO. 
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9.4  TECHNOLOGIES  EXPLORED 

9.4.1  Reconnaissance  Environment 

The  reconnaissance  environment  consisted  of  a  large  room  (about  60  m2)  subdivided  in  several  sections,  and  a 
smaller  adjacent  room  (about  8  m2).  Eleven  possible  target  objects  varying  in  size  were  positioned  at  different 
height  levels  in  the  reconnaissance  environment: 

A)  Soda  can  bomb  on  a  table; 

B)  Hand  grenade  on  the  ground; 

C)  Soda  can  bomb  on  the  ground; 

D)  Hand  grenade  near  the  ceiling; 

E)  Semtex  on  the  ground; 

F)  Bomb  shell  on  a  table; 

G)  Pipe  bomb  on  the  ground; 

H)  Semtex  with  timer  on  a  chair; 

I)  Mine  on  a  water  container; 

J)  Land  mine  on  a  high  cupboard  shelf;  and 

K)  Land  mine  on  a  high  cupboard  shelf. 

Objects  B,  D,  E,  G,  H,  J  were  used  as  targets;  the  others  were  used  as  decoy  targets  or  practice  targets. 

9.4.2  Control  Station 

The  control  station  was  located  in  a  tent  next  to  the  building  of  the  reconnaissance  environment  (see  Figure  9-1). 
The  control  station  consisted  of  a  user  interface  with  a  NVIS  nVISOR  Head-Mounted  Display  (either  stereo  or 
mono,  depending  on  the  experimental  condition),  an  Xsens  MTi  motion  sensor  as  a  headtracker,  stereo 
headphones,  and  a  Logitech  Dual  Action  game  controller.  Three  human-robot  interface  setups  of  the  control 
station  were  used  in  this  experiment,  as  explained  in  the  section  below  on  Experimental  Setup. 
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Figure  9-1 :  Reconnaissance  Area  and  Control  Station. 


9.4.3  Unmanned  System 

The  Unmanned  Ground  Vehicle  (UGV)  used  was  TNO’s  robot  called  ‘GeneraaT.  This  UGV  is  a  fully 
manually  controlled  UGV,  with  a  fast  and  powerful  pan-tilt-roll  sensor  system  that  can  accurately  mimic 
human  head  movements  enabling  remote  perception  of  the  UGV  environment. 


9.5  HUMAN  FACTORS  ISSUES  EXPLORED 

Our  main  human  factors  research  questions  for  the  current  experiment  were: 

•  Does  headtracking  control  lead  to  improved  performance  as  compared  to  joystick  control? 
(comparison  between  Mono  Headtracking  and  Mono  Joystick  human-robot  interfaces,  see  description 
below); 

•  Does  a  3D  audio  system  lead  to  improved  performance  as  compared  with  a  directional  microphone? 
(comparison  between  Mono-Headtracking  and  Telepresence  human-robot  interfaces);  and 

•  What  would  be  the  maximum  performance  benefit  of  telepresence  functionality  (with  headtracking 
and  stereo  sensor  information)  as  compared  with  the  currently  mostly  used  control  systems  with 
joystick  control  and  mono  sensor  information?  provided  it  exists?  (comparison  between  Telepresence 
and  Mono-Joystick  human-robot  interfaces). 

For  answering  these  questions  we  considered  the  quality  of  performance  in  locating  and  identifying  objects  in 
an  indoor  audio  detection  task,  in  three  experimental  conditions  for  the  user  interfaces: 
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•  Mono-Joystick:  Mono  audio  and  video  on  Head  Mounted  Display,  with  joystick  control  for  robot 
movements  and  heading  of  sensor  system.  Participants  were  asked  (and  reminded  when  needed)  not 
to  move  their  heads. 

•  Mono-Headtracking:  Mono  audio  and  video  on  Head  Mounted  Display,  with  joystick  control  for 
robot  movements  and  headtracking  for  directing  the  sensor  system. 

•  Telepresence:  Stereo  audio  and  video  on  Head  Mounted  Display,  with  joystick  control  for  robot 
movements  and  headtracking  for  directing  the  sensor  system.  We  refer  to  this  configuration  as 
Telepresence. 

Each  participant  performed  the  sound  detection  task  18  times.  Each  of  the  six  targets  was  used  for  each  of 
three  conditions.  After  each  trial,  the  participant  switched  to  one  of  the  other  two  experimental  conditions. 


9.6  UNMANNED  SYSTEMS  USED 

The  Unmanned  Ground  Vehicle  (UGV)  used  was  TNO’s  robot  called  ‘Generaal’.  This  UGV  has  been  used  in 
prior  studies  in  our  lab  [2].  It  is  a  fully  manually  controlled  UGV,  with  a  fast  and  powerful  pan-tilt-roll  system 
that  can  accurately  mimic  human  head  movements.  On  top  are  two  cameras  for  providing  stereo  vision  at  the 
control  station,  and  two  microphone  arrays  that  can  be  positioned  at  either  side  for  spatial  3D  audio,  or  next  to 
each  other  in  front  thereby  functioning  as  a  directional  microphone.  The  horizontally  positioned  red-tipped 
pointer  in  front  of  the  vehicle  was  the  reference  point  for  the  participants  in  approaching  the  target  as  closely 
as  possible. 
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Figure  9-2:  TNO’s  Unmanned  Vehicle  ‘GeneraaP.  Left  panel  shows  the  vehicle  with  sensor  unit  on  a 
pan-tilt-roll  motion  platform  with  3D  audio  and  stereo  visual  sensors.  The  sensor  unit  is  presented 
enlarged  in  the  upper  right  panel,  with  the  microphone  array  placed  in  their  3D  audio  position,  at 
either  side  of  the  stereo  cameras.  The  lower  right  panel  shows  how  the  two  microphone  arrays  were 
placed  in  the  centre  position  right  above  the  stereo  cameras,  for  receiving  directional  mono  sound. 


9.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 


Planning/Design 

Execution 

Analysis 

Communication 

TNO  and  US 

Army 

TNO  and  US 

Army 

TNO  and  US 

Army 

Coordination 

TNO  and  US 

Army 

TNO  and  US 

Army 

TNO  and  US 

Army 

Collaboration 

TNO  and  US 

Army 

TNO  and  US 

Army 

TNO  and  US 

Army 
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9.8  SUMMARY  OF  TECHNICAL  DEMONSTRATION  RESULTS 

Non-parametric  Wilcoxon  Matched  Pairs  tests  show  that  the  percentage  correct  target  ID  in  the  Telepresence 
condition  is  significantly  higher  (87.5%)  than  in  both  Mono-Joystick  (61.5%;  p  <  .05)  and  Mono- 
Headtracking  (64.6%;  p  <  .005);  Mono-Headtracking  and  MJ  do  not  differ  (p  =  .51). 

The  repeated  measures  ANOVA  on  time  to  target  identification  indicates  a  main  effect  for  Human-Robot 
Interface  (F(2,30)  =  17.48,  p  <  .001);  all  Tukey  HSD  post  hoc  tests  were  significant  (all  p  <  .05).  Time 
to  target  identification  is  shortest  for  Telepresence  (65.0  seconds),  followed  by  Mono-Headtracking 
(88.8  seconds),  and  longest  for  Mono-Joystick  (1 13.5  seconds). 


9.9  LESSONS  LEARNED 

Including  head  motion  tracking  for  controlling  a  directional  microphone  significantly  improves  a  human 
operator’s  detection  and  localization  of  audio  targets  in  a  reconnaissance  mission.  This  performance  is  boosted 
even  more  when  human  natural  listening  behavior  is  further  mimicked  when  3D  audio  is  presented  using 
advanced  microphone  arrays  instead  of  mono  audio  generated  by  a  directional  microphone. 

We  have  learned  that  field  tests  are  valuable  if  not  crucial  in  estimating  the  possible  operational  benefits  of 
technology  that  already  has  been  tested  and  improved  in  the  laboratory  conditions. 


9.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  findings  of  this  study  are  limited  to  indoor  reconnaissance  in  which  no  other  sounds  are  present  except  for 
the  audio  target. 


9.11  CONCLUSIONS 

In  Section  9.5  we  identified  three  research  questions  that  can  be  answered: 

•  Does  headtracking  control  lead  to  improved  performance  as  compared  to  joystick  control?  The  results 
show  no  difference  between  the  Mono-Headtracking  condition  and  the  Mono-Joystick  condition  in 
correctness  of  target  identification.  However,  with  joystick  control,  more  time  is  needed  for  target 
identification:  about  26%  more  time  is  needed  when  using  a  joystick  for  sensor  control  (here  Mono- 
Joystick  with  1 1 1.2  seconds  on  average)  as  compared  to  headtracking  (here  Mono-Headtracking  with 
88.2  seconds). 

•  Does  a  3D  audio  system  lead  to  improved  performance  as  compared  with  a  directional  microphone? 
When  comparing  the  Telepresence  condition  (having  3D  audio)  with  the  Mono-Headtracking 
condition  (having  a  directional  microphone),  we  see  that  with  Telepresence  the  percentage  of 
correctly  identified  targets  is  about  23%  higher.  In  addition,  target  identification  takes  about  35% 
more  time  without  having  the  3D  audio  functionality  available  (here  88.2  and  65.0  seconds  for  Mono- 
Headtracking  and  Telepresence  respectively). 

•  What  would  be  the  maximum  performance  benefit  of  telepresence  functionality  as  compared  with  the 
currently  mostly  used  control  systems  with  joystick  control  and  mono  sensor  information,  provided  it 
exists?  Based  on  the  results  in  this  study,  the  use  of  a  Telepresence  human-robot  interface  results  in 
identification/localization  times  for  audio  that  are  about  42%  shorter  than  with  current  commonly 
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used  interfaces  (65.0  sec  and  111.2  sec  for  Telepresence  and  Mono-Joystick  respectively). 
In  addition,  the  target  identification  performance  increases  by  about  26%  when  using  the 
Telepresence  human-robot  interface. 

These  promising  results  encourage  more  elaborate  testing  in  operational  settings,  following  our  initial  field 
trials  with  telepresence  UGV  control  reported  in  [3]. 


9.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

In  continuing  the  collaboration  between  TNO  and  ARL,  we  are  considering  two  options  that  could  be 
performed  in  parallel.  First,  we  believe  that  telepresence  could  be  even  more  beneficial  if  other  multi-modal 
user  interface  are  included  as  well,  in  particular  vibrotactile  interfaces  (e.g.,  for  indicating  direction  of 
movement,  the  next  waypoint,  collision  warnings  for  obstacles).  Second,  we  plan  to  investigate  the  extent  to 
which  performance  in  a  reconnaissance  mission  could  further  increase  by  combining  telepresence  operator 
involvement  with  robot  autonomy  in  a  well-designed  adaptive  automation  concept. 
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10.1  DATES 

24  January  2011-4  February  2011. 


10.2  LOCATION 

OTA,  Lisbon,  Portugal. 


10.3  SCENARIO/TASKS 

At  the  Underwater  Systems  and  Technology  Laboratory  (LSTS)  [1]  we  have  been  designing,  building  and 
operating  a  number  of  heterogeneous  unmanned  vehicles.  These  include  Remotely  Operated  Vehicles  (ROV) 
[2],  Autonomous  Underwater  Vehicles  (AUV)  [3], [5], [6],  and  Autonomous  Surface  Vehicles  (ASV)  [4]. 
We  have  been  also  developing  UAVs  [7]  as  a  result  of  our  collaboration  with  the  Portuguese  Air  Force 
Academy. 

Recent  technological  advances  led  to  the  creation  of  very  capable  unmanned  systems  constructed  using  low 
cost  hardware.  This  allows  the  application  of  these  technologies  to  scenarios  where  multiple  unmanned 
systems  can  be  employed  simultaneously  like  patrolling,  adaptive  sensing,  search  and  rescue,  etc.  However, 
human  operators  have  turned  into  an  increasingly  scarcer  and  more  expensive  resource  whose  exploitation 
shall  be  optimized. 

In  this  chapter,  we  describe  a  conceptual  framework  for  optimal  inclusion  of  the  operator  in  the  control  loop 
and  the  application  of  these  concepts  into  a  Command  and  Control  (C2)  operator  interface.  Our  objective  is  to 
distribute  and  reduce  the  workload  of  a  decentralized  team  of  operators  controlling  multiple  UAVs. 
To  achieve  this  goal  we  intend  to  advise  operator’s  actions  and  reconfigure  C2’s  layout  using  an  automated 
methodology.  The  operator  can  have  different  levels  of  situation  awareness,  at  different  stages  of  the  mission. 
The  system  will  help  operators  to  dynamically  configure  an  optimal  view  of  the  mission  state  from  a  set  of 
predefined  console  layout  profiles. 

We  interpreted  and  adapted  the  original  (Level  Of  Autonomy)  LOA  matrix  (Table  10-1)  into  our  framework 
for  optimal  inclusion  of  the  operator  in  the  control  loop.  The  LOA-Level  of  Autonomy  Table  [10]  is  based  on 
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Sheridan’s  10-level  of  autonomy  scale  [11]  and  simplified  to  present  only  eight  levels  of  autonomy.  The  two 
dimensions  of  the  matrix  (Table  10-1)  are  the  eight  levels  (matrix  rows)  crossed  with  four  functional 
categories  (matrix  columns).  The  second  dimension  presented  in  this  matrix  is  the  division  of  each  task  into 
four  functional  steps.  These  tasks  present  human  decision-making  processes  as  a  set  of  OODA  cycles 
(Observe,  Orient,  Decide,  and  Act). 

Table  10-1:  Partial  LOA  Matrix  as  Originally  Published  in  [10]. 


Level  Observe  Orient  Decide  Action 


8 


The  computer  gathers, 
filters,  and  prioritizes 
data  without  displaying 
any  information  to  the 
human. 


The  computer  gathers, 
filters,  and  prioritizes 
data  without  displaying 

any  information  to  the 
human.  Though,  a 
“program  functioning” 
flag  is  displayed. 


The  computer 
predicts,  interprets, 
and  integrates  data 
into  a  result  which  is 
not  displayed  to  the 
human. 

The  computer 
analyses,  predicts, 
interprets,  and 
integrates  data  into  a 
result  which  is  only 
displayed  to  the 
human  if  result  fits 
programmed  context. 


The  computer 
performs  ranking 
tasks.  The  computer 
performs  final 
ranking,  but  does  not 
display  results  to  the 
human. 

The  computer 
performs  ranking 
tasks.  The  computer 
performs  final  ranking 
and  displays  a  reduced 
set  of  ranked  options. 

Without  displaying 
“why”. 


Computer  executes 
automatically  and 
does  not  allow  any 
human  interaction. 


Computer  executes 
automatically  and 
only  informs  the 
human  if  required  by 
context.  It  allows  for 
override  ability  after 
execution.  Human  is 
for  shadow 
contingencies. 


1 


Human  is  the  only  Human  is  responsible 
source  for  gathering  and  for  analyzing  all  data, 
monitoring  (defined  as  making  predictions 

filtering  and  and  interpretation  of 

prioritizing)  all  data.  the  data. 


The  automate  does  not 
assist  in  or  perform 
ranking  tasks.  Human 
must  do  it  all. 


Human  alone  can 
execute  decision. 


Table  10-2  is  used  to  categorize  the  operator  skills  using  the  LOAs  he  is  certified  to  respond  to,  the  CP 
(Console  Profile)  the  operator  is  familiarized  and  the  number  of  vehicles  he  can  handle  safety  at  a  certain 


LOA. 


Table  10-2:  Fields  Used  to  Infer  About  the  Operators  Skills  in  the  Framework. 


Certified  Type  of  LOA  Certified  Consoles  Profiles  Number  of  Vehicles 


Type  of  manoeuvre  the 
operator  is  certified. 


Set  of  operation  Consoles  the  Operator  fan-out  of  vehicles 
operator  is  familiarized.  By  (for  one  LOA) 

preference  order,  (for  one  LOA) 
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To  exemplify  the  framework’s  execution  we  will  evaluate  a  mission  scenario  where  the  operators  have  to  find 
a  target  and  follow  it.  There  will  be  two  operators  and  five  UAVs  in  this  scenario. 

Currently  existing  UAVs  offer  little  adaptability  in  terms  of  automation:  operators  can  command  the  UAV  to 
fly  autonomously,  following  a  pre-de fined  flight  path,  or  they  can  control  it  manually.  For  this  example  we 
will  use  2  LOAs  for  the  operators,  and  another  one  of  full  autonomy  used  in  handover  and  in  emergency 
situations.  The  operators  LOAs  to  be  used  are  further  sub-divided  into  a  high  level  control  LOA  and  low  level 
control  LOA  in  this  scenario. 

All  three  LOAs  used  are  described  as  follows: 

•  Operational  Mode  1  -  Tele-Operation  or  Direct  Control  -  LOA  =  (3, 2, 2, 2); 

•  Operational  Mode  2  -  Survey  -  LOA  =  (6, 6, 7, 6);  and 

•  Operational  Mode  3  -  Full  Autonomy  -  LOA  =  (8, 8, 8, 8). 

The  matrix  represented  in  Table  10-1  can  be  related  with  the  creation  of  different  types  of  console  profiles. 
Different  console  profiles  can  be  associated  to  different  combinations  of  the  four  functional  categories 
(OODA)  -  operational  modes.  For  the  presented  framework  we  have  a  direct  relation  of  LOA  and  CP. 
The  formal  representation  for  CP-LOA  tuple  is: 


CP-LOA  =  ({Obsi...Obsn},{Orii...Orin},{  Deci...Decn},  {Acti...Actn  }) 


The  elements  on  the  tuple  are  represented  as  sets  so  we  can  group  the  OODA  functional  categories.  This  way 
it  is  possible  to  have  one  CP  capable  of  handling  different  Operational  Modes. 

We  will  use  two  CPs  (CPI  =  ({3}, {2}, {2}, {2})  and  CP2  =  ({6-7}, {6-7}, {6-7}, {6-7}) )  to  handle  this  mission 
example  as  follow: 
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Figure  10-1:  Two  Console  Profiles  Used  in  Mission  (For  Low  and  High  Level  Control). 


For  this  mission  example  we  will  have  two  operators  with  the  following  Skills  Tables. 
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Table  10-3:  Skills  Table  -  Operator  1  Can  Handle  3  UAVs  in  High  Level  Control  and 
1  UAV  in  Low  Level  Control;  Operator  2  Can  Handle  4  UAVs  in  High  Level  Control. 


Certified  Type  of  LOA 

Certified  CPs  (Consoles  Profiles) 

Number  of  Vehicles 

Operator  1 

(3, 2, 2, 2) 

{CPI} 

1 

(6, 6, 7, 6) 

{CP2} 

3 

Operator  2 

(6, 6, 7, 6) 

[CP2} 

4 

Figure  10-2  illustrates  the  5  most  important  steps  taken  when  one  of  the  operators  finds  the  target.  The  state  of 
the  system  before  any  of  the  operators  finds  the  target  is  the  beginning  step  (step  1)  of  Figure  10-2.  Initially, 
all  the  UAVs  are  in  survey  mode  -  mode  2  of  our  LOA  definition.  Both  of  the  operators  are  using  CP2  to 
control  the  UAVs:  define  survey  areas  and  look  at  part  of  the  payload  data  (video). 


Figure  10-2:  Logic  of  Operation  for  Workload  Distribution  Mission  Example. 
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In  step  2  of  Figure  10-2,  Operator  1  finds  the  target.  The  target  must  be  followed  using  direct  control.  To  solve 
the  excessive  workload  of  Operator  1  (Operator  1  can  handle  only  1  UAV  in  Operational  Mode  1  -  Tele¬ 
operation  -  and  Operator  2  is  not  certified  for  Operational  Mode  1,  consult  Table  10-3),  the  system  (mission 
supervisor)  will  try  to  assign  this  UAV  in  mode  1  -  Tele-Operation  -  to  some  operator.  The  only  operator 
capable  of  handling  mode  1  is  operator  1,  as  defined  in  Table  10-3.  Since  the  operator  1  is  capable  of  handling 
only  one  UAV  in  this  Mode,  the  mission  supervisor  will  advise  Operator  1  to  hand-over  the  other  2  UAVs 
from  Operator  1  to  Operator  2.  Here  starts  step  3  with  the  handover  process:  Operator  1  releases  the  two 
controlled  UAVs  by  setting  them  at  mode  3  (Full  Autonomy). 

Finally,  in  step  5,  Operator  2,  that  has  accepted  the  hand-hover,  takes  over  these  UAVs  which  are  on  Mode  2 
and  the  Operator  1  can  now  handle  Mode  1  (Tele-Operation)  and  follow  the  target.  In  this  step  the  Mission 
Supervisor  advises  Operator  1  to  use  CPI -Tele-operation  to  respond  mode  1  LOA,  which  requires  full 
attention  to  the  vehicle,  according  to  his  skills  in  Table  10-3. 


10.4  TECHNOLOGIES  EXPLORED 

The  concepts  of  operation  for  multi -UAV  teams  differ  from  single  UAVs  in  the  sense  that  in  the  former  there 
exist  common  objectives  like  maintaining  a  common  knowledge  database  [8]  and  redundant  execution  of 
crucial  actions  [9]. 

In  our  C2  framework,  UAVs  can  be  tasked  either  individually  by  an  operator  or  they  can  be  tasked  by  a 
software  agent  that  acts  as  an  operator  (Team  Supervisor).  The  team  supervisor  divides  work  among  the 
vehicles  according  to  a  multi -UAV  mission  specification  and  simple  task-allocation  algorithms.  If  the  control 
over  the  UAV  is  not  overridden,  they  carry  out  planned  behavior  until  they  are  faced  with  failures,  or  there  are 
any  other  unpredicted  situations  in  which  they  contact  the  ground  station  and  require  human  intervention. 

To  provide  system-level  control  of  multiple  vehicles,  we  use  a  software  agent  that  holds  a  multi-UAV  mission 
specification.  This  mission  specification  is  currently  a  list  of  individual  plans  that  need  to  be  executed  by 
UAVs.  Tasks  are  divided  among  UAVs  in  a  way  that  workload  is  shared  among  capable  vehicles.  Some  tasks 
however  also  require  the  intervention  of  human  operators  for  correct  execution,  so  the  availability  of  operators 
must  be  taken  into  account  by  the  team  supervisor  while  tasking  the  network. 

As  stated  before,  this  framework  was  employed  in  an  existing  C2  software  framework:  Neptus.  Neptus  has 
an  underlining  architecture  that  provides  the  means  for  creating  the  various  consoles  used  in  different  CPs. 
This  section  introduces  Neptus  and  gives  an  example  of  such  consoles. 

Neptus  is  a  distributed  C2  framework  for  operations  with  networked  vehicles,  systems,  and  human  operators. 
Neptus  supports  all  the  phases  of  a  mission’s  life  cycle:  planning,  simulation,  execution,  and  post-mission 
analysis.  Neptus  supports  concurrent  operations.  Vehicles,  operators,  and  operator  consoles  come  and  go. 
Operators  are  able  to  plan  and  supervise  missions  concurrently  [12]. 

In  Neptus,  the  Console  Builder  application  facilitates  the  addition  of  new  vehicles  with  new  sensor  suites  to 
Neptus.  In  this  application  the  operator  can  build,  configure,  and  save  vehicle  consoles.  There  are  two 
important  aspects  for  console  configuration:  visual  components  and  event  communications.  The  internal 
Neptus  event  communication  system  is  based  on  a  tree  structure  (following  the  blackboard  design  pattern 
[13]),  where  nodes  indicate  the  subject  of  data  values  in  leafs.  Neptus  visual  components  can  become  listeners 
of  a  single  variable  (tree  leaf)  or  of  a  defined  variable  domain  (tree  branch).  Whenever  a  message  arrives, 
using  the  IMC  [14]  communication  protocol,  that  data  is  stored  in  a  specific  tree  branch  and  listeners  for  any 
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branch  that  encloses  the  affected  branch/leaf  are  informed  of  the  incoming  network  data.  In  a  similar  way, 
output  data  is  sent  to  the  network  by  Neptus  console  components  through  the  variable  tree.  The  variable  tree 
system  is  also  used  for  event  communication  between  Neptus  console  components. 

There  are  two  states  in  the  Neptus  generic  console  builder  application:  editing  and  operational.  In  editing 
mode,  the  palette  of  available  components  (STANAG  planning  panel,  compass  panel,  Tenderer  panel,  video 
panel)  becomes  visible.  Users  can  then  add  and  place  components  freely  inside  console  main  panel. 
Component  properties  can  be  edited  to  connect  the  panels  to  different  systems  and  variables.  When  all 
components  are  ready,  correctly  placed  and  connected  to  the  system  variable  tree,  the  user  can  switch  the  state 
of  the  application  to  the  Operational  mode.  In  this  mode,  the  position  of  the  components  in  the  console  is  fixed 
and  it  responds  to  the  user  interactions  (Figure  10-3). 


Neptus  Framework 


Figure  10-3:  Neptus  Internal  Communications  System. 


Besides  having  the  capability  of  dynamically  creating  new  consoles  during  a  specific  mission,  Neptus  also  has 
predefined  consoles  already  available  for  the  LOA  switches  the  presented  framework  requires.  These  consoles 
go  from  standard  tele-operation  consoles,  as  seen  as  example  1  of  Figure  10-4,  to  supervision  consoles, 
as  seen  on  the  right  (example  2)  of  Figure  10-4.  These  consoles  have  different  layouts  depending  on  the 
central  function  they  have.  For  instance  a  tele-operation  console  will  typically  have  more  detailed  data  about 
the  UAV  under  its  control,  whereas  a  supervision  console  will  only  have  a  simplified  view  of  the  current  UAV 
to  allow  a  broader  view  of  the  whole  team.  As  an  example  of  said  consoles  we  introduce  the  details  behind  the 
current  flight  manager  console  used  for  UAV  mission  supervision  at  the  LSTS. 
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Figure  10-4:  Tele-Operation  (Low  Level  Control  -  1)  and 
Supervisory  Control  (High  Level  Control  -  2)  Consoles. 

The  supervisory  control  console,  as  seen  in  Figure  10-4,  was  developed  based  on  a  Real-Time  Strategy  (RTS) 
paradigm  with  the  intent  of  applying  the  concepts,  learned  by  this  type  of  games,  on  how  to  efficiently  control 
and  supervise  groups  of  units  of  various  dimensions  and  with  varying  capabilities.  This  approach,  while  not 
being  new,  has  allowed  the  implementation  of  a  console  which  supports  high  LOA  levels  CP-LOA  = 
({6-7}, {6-7},  {6-7},  {6-7})  while,  at  the  same  time,  enables  the  supervision  of  UAV  teams  with  a  low 
workload  rating  value  for  the  operators. 

10.5  HUMAN  FACTORS  ISSUES  EXPLORED 

The  two  main  human  factors  that  we  explore  in  this  framework  is  situation  awareness  and  workload  index. 
To  that  extent,  we  perform  systematic  evaluations  of  these  metrics  through  the  use  of  NASA’s  TLX  method, 
for  workload  analysis  and  the  SAGAT  method  for  situation  awareness  testing.  An  example  of  the  workload 
values  collected  in  one  of  these  tests  can  be  viewed  in  Figure  10-5. 
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Figure  10-5:  Flight  Manager  Console’s  Total  Workload 
Rating,  Using  NASA-TLX  [15],  in  a  3  UAV  Scenario. 
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ORGANIZATION 


10.6  UNMANNED  SYSTEMS  USED 

For  the  various  flights  performed  a  vast  array  of  unmanned  systems  were  used.  Some  of  these  systems  can  be 
viewed  in  Figure  10-6. 


Figure  10-6:  UAVs  ANTEX  X02  1-4  Series. 


Due  to  maintenance  reasons,  it  is  for  us  very  common  to  change  UAV  models  in  the  middle  of  a  test  run. 
Nevertheless  we  present  specific  details  of  the  systems  which  are  more  regularly  used  on  the  shakedown  tests. 

Table  10-4:  Specific  Data  of  the  ANTEX  X02  -  03  UAVs. 


ANTEX  X02  Series 

Max  Weight 

10  KG 

Width 

2,4  m 

Max  Speed 

150  Km/h 

Max  Payload 

4  Kg 

Max  Autonomy 

5  h 

Max  Altitude 

2  Km 
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10.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 


Planning/Design 

Execution 

Analysis 

Communication 

X 

X 

X 

Coordination 

Collaboration 

10.8  SUMMARY  OF  TD  RESULTS 

We  presented  the  concepts  behind  a  framework  for  managing  UAV  task  and  workload  allocation  between 
various  operators  in  a  mission  scenario.  This  framework  was  applied  to  the  development  of  a  Command  and 
Control  (C2)  application  which  is  capable  of  self-adaptation,  operator  advisement  and  automatic  task 
distribution  among  operators  and  UAVs  according  to  mission  objectives,  phase  and  occurrences.  An  example 
scenario  of  this  framework,  as  well  as  an  example  of  the  details  around  one  of  the  consoles  used  by  the 
operators,  was  presented  and  discussed. 


10.9  LESSONS  LEARNED 

This  C2  application  enables  a  clear  view  and  presence  on  the  remote  environment  by  putting  the  operator 
much  closer  to  the  control  loop,  whether  it  is  high  level  or  low  level  control,  with  the  consequent  improved 
redistribution  of  tasks  and  situational  awareness.  NASA  Task  Load  Index  (TLX)  was  used  as  a  means  to 
determine  the  adequacy  of  the  C2  interface  and  functionalities.  The  preliminary  results  obtained  with  this 
framework  are  promising  and  we  are  confident  that  its  use  will  vastly  improve  the  reliability  of  multi-UAV 
teams  by  augmenting  their  compatibility  with  more  mission  scenarios. 


10.10  STUDY  CONSTRAINTS/LIMITATIONS 

Although  we  have  a  vast  base  of  operations  and  a  large  array  of  working  vehicles  to  test  this  framework, 
the  majority  of  the  testing  took  place  in  a  simulated  environment.  This  will,  on  one  hand,  allow  us  to  detect 
faults  in  the  system  without  endangering  our  vehicles  but,  on  the  other  hand,  limit  the  conclusions  that  can  be 
extrapolated  from  the  experiments  due  to  lack  of  realism. 


10.11  CONCLUSIONS 

Throughout  this  chapter  we  referenced  the  growing  importance  of  multi-UAV  systems,  paying  special 
attention  to  the  need  of  optimal  in-the-loop  inclusion  of  operators  for  the  successful  use  of  these  systems. 
The  tested  framework  for  managing  UAV  task  and  workload  allocation  between  various  operators  in  a 
mission  scenario  proved  to  be  an  improvement  over  the  previous  approach.  Further  testing,  with  simulated 
scenarios,  will  provide  new  insights  over  the  real  capabilities  of  the  proposed  framework. 
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10.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

We  intend  to  implement  another  formal  layer  over  the  presented  framework  to  extract  the  viability  of  mission 
execution.  It  is  possible  to  reach  combinations  of  plan  state  manoeuvres  that  overload  the  response  of  the 
operator’s  team.  Our  approach  will  use  Petri  Nets  for  the  model  to  tackle  this  issue  (another  similar  example 
can  be  consulted  in  [16]).  By  studding  the  plan  loaded  in  each  UAV  and  applying  a  transformation  that 
combines  all  UAV  plan  states,  the  plan  state  change  events  probabilities,  and  the  operators  team  recourses  into 
a  Petri  Net,  we  can  infer  about  the  probability  of  reaching  a  failure  state.  The  failure  state  can  be  considered  to 
be  a  state  where  operator  resources  do  not  correspond  to  the  mission  state  demands.  In  the  last  analysis, 
we  can  know  the  probability  of  reaching  one  mission  state  before  the  Mission  Team  Supervisor  has  to  process 
the  resource  allocation.  This  information  can  be  used  to  optimize  the  resource  allocation  process  and  also  to 
help  avoiding  some  mission  states  in  the  mission  planning  phase  (e.g.,  find  and  avoid  states  that  require  full 
autonomy  LOA  =  (8, 8, 8, 8)  manoeuvres). 
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11.1  DATES 

1-30  November  2006. 


11.2  LOCATION 

FOI,  Linkoping,  Sweden. 


11.3  SCENARIO/TASKS 

The  operator’s  task  was  to  navigate  one,  two,  or  three  partly  autonomous  UGVs  to  pre-designated  inspection 
points  in  a  simulated  urban  environment.  The  UGVs  were  mainly  manually  controlled  and  had  only  a  limited 
autonomous  function  in  that  they  could  maintain  the  current  heading  and  velocity  while  unattended  to  by  the 
operator.  Figure  11-1  shows  an  example  of  the  operator’s  control  station  where  the  position  of  the  UGVs  and 
the  shape  and  colour  coded  inspection  points  are  indicated  on  the  map  in  the  upper  right  corner.  The  colour 
coded  3D-objects  representing  the  inspection  points  were  shown  when  within  view  of  the  UGV’s  camera. 
The  task  was  to  navigate  each  UGV  to  the  inspection  point  with  the  corresponding  colour.  A  new  inspection 
point  was  shown  when  the  UGV  reached  the  indicated  inspection  point.  The  operators  alternated  control 
sequentially  between  the  UGVs  to  manually  change  the  heading  and  velocity  and  engage  the  autonomous 
function  to  maintain  the  last  control  action  while  the  operator  attended  to  the  other  UGVs.  Performance  was 
measured  by  the  number  of  inspection  points  that  were  reached  within  a  10-minute  period. 
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Figure  11-1:  Example  of  the  Operator’s  Control  Station.  The  task  was  to  navigate  the  blue  UGV 
to  the  blue  inspection  point,  the  red  UGV  to  red  inspection  point,  and  the  green  UGV 
to  the  green  inspection  as  indicated  on  the  map  in  the  upper  right  corner. 


11.4  TECHNOLOGIES  EXPLORED 

Only  a  limited  autonomous  function  was  used  that  enabled  the  UGVs  to  maintain  the  current  heading  and 
velocity. 


11.5  HUMAN  FACTORS  ISSUES  EXPLORED 

One  reason  why  it  is  so  difficult  to  improve  the  operator  to  vehicle  ratio  and  enable  one  or  a  few  operators  to 
control  several  robots  is  that  the  attentional  requirements  are  so  high  for  controlling  even  one  robot  that  any 
additional  robots  overload  the  operator  or  operators  and  hampers  the  performance.  The  purpose  of  the  study 
was  to  investigate  these  attentional  requirements  for  a  typical  ground  robot  when  performing  a  basic 
navigation  task  in  a  military  setting,  and  to  what  extent  a  limited  autonomous  function  was  useful  in 
facilitating  control  of  several  robots. 
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An  exploratory  performance  measure  called  Instantaneous  Performance  was  also  used  to  measure 
performance  continuously  during  a  trial  rather  than  only  at  the  end  of  a  trial  by  the  number  of  inspection 
points  that  were  reached  [2].  The  Instantaneous  Performance  is  computed  by  normalizing  the  UGVs’  current 
velocity  and  heading  relative  the  shortest  path  towards  the  next  inspection  point  and  the  maximum  velocity. 
This  results  in  measure  where  1  means  that  the  UGV  is  moving  along  the  shortest  path  towards  the  inspection 
point  at  maximum  velocity  and  -1  that  the  UGV  moving  in  the  opposite  direction  at  maximum  velocity. 
The  benefit  of  Instantaneous  Performance  is  that  is  allows  an  assessment  of  operators’  control  strategies. 
Please  see  Lif  et  al.  [1]  for  more  information  about  the  study. 

11.6  UNMANNED  SYSTEMS  USED 

One,  two,  or  three  simulated  UGVs. 

11.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

The  results  of  the  study  have  been  presented  at  Task  Group  meetings,  as  well  as  at  a  conference  session 
arranged  by  a  Task  Group  member  [1].  The  following  table  summarizes  the  extent  of  the  NATO  collaboration. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

Coordination 

Collaboration 

11.8  SUMMARY  OF  TD  RESULTS 

The  results  show  that  the  operators  reach  30%  more  inspection  points  when  using  two  UGVs  compared  to 
when  only  using  one  UGV.  Adding  a  third  UGV  did  not  provide  any  additional  improvement  in  performance, 
however.  Due  to  mental  overload  when  using  more  than  one  UGV,  the  UGVs  stand  still  30%  of  the  time  when 
using  two  UGVs  and  50%  of  the  time  when  using  three  UGVs.  Furthermore,  the  duration  of  the  stand  stills 
last  between  5  seconds  and  1  minute,  which  is  undesirable  in  a  potentially  hostile  environment.  The  mental 
overload  was  also  evident  in  the  operators’  control  strategies  where  some  operators  aimed  the  UGVs  towards 
walls  to  know  where  to  find  the  UGV  when  they  regained  control,  or  even  completely  abandoning  UGVs. 
Overall,  the  operators  control  the  UGVs  manually  90%  of  the  time  although  they  use  the  autonomous  function 
more  when  controlling  more  UGVs.  The  correlation  between  Instantaneous  Performance  and  the  number  of 
reached  inspection  points  was  .95,  which  shows  that  it  is  valid  performance  measure.  The  operators’  control 
strategies  were  not  explored  further,  however. 

11.9  LESSONS  LEARNED 

Typical  issues  in  control  of  multi-robot  systems  can  be  investigated  in  a  laboratory  environment  without 
experienced  operators. 
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11.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  operators  did  not  have  any  previous  experience  in  controlling  UGVs  and  were  thus  rather  na'ive.  The  lack 
of  experience  was  evident  in  that  some  subjects  had  difficulty  operating  the  UGVs  in  a  safe  way,  particularly 
when  controlling  more  than  one  UGV. 


11.11  CONCLUSIONS 

The  results  show  that  the  limited  autonomous  function  was  insufficient  to  significantly  improve  the  operator 
to  vehicle  ratio.  The  operators  are  saturated  even  when  only  controlling  two  UGVs  in  a  basic  navigation  task. 
More  advanced  partly  autonomous  functions  or  control  station  interfaces  that  reduce  the  attentional 
requirements  are  therefore  necessary  to  improve  the  operator  to  vehicle  ratio. 


11.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

Although  research  programs,  such  as  the  DARPA  Grand  Challenge,  show  that  ground  robots  can  navigate 
autonomously  in  an  uncertain  environment,  such  technologies  are  typically  not  available  for  UGVs  that  are 
used  for  tactical  reconnaissance.  An  alternative  approach  is  therefore  to  develop  better  interfaces  that  reduce 
the  attentional  requirements  and  improve  the  operators’  strategies  for  sequentially  alternating  the  control 
between  robots.  One  approach  to  develop  such  interfaces  is  to  derive  a  prioritization  order  for  relevant  goal 
attainment  states  from  either  simulated  or  empirical  data  [3].  This  prioritization  order  can  then  be  used  to 
indicate  which  UGV  that  is  in  most  need  of  service.  There  are  currently  no  plans  for  future  research,  however. 
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12.1  DATES 

1  December  2007  -  28  February  2008. 


12.2  LOCATION 

FOI,  Linkoping,  Sweden. 


12.3  SCENARIO/TASKS 

The  operator’s  task  was  tactical  reconnaissance  along  the  route  of  advance  for  a  convoy  to  locate  and 
neutralize  mobile  threats  in  a  simulated  urban  environment.  The  convoy  consisted  of  15  vehicles  that 
advanced  along  a  fixed  route  that  was  about  1.7  km  long.  Along  the  convoy’s  route  of  advance  were  28 
mobile  threats  that  either  crossed  or  came  near  the  route  of  advance.  The  operators  performed  the  tactical 
reconnaissance  using  either  six  or  twelve  UGVs  that  autonomously  searched  for  the  mobile  threats  within 
designated  areas,  as  well  as  detected  and  tracked  the  mobile  threats.  The  operator  controlled  the  UGVs  by 
allocating  them  into  search  group  and  designating  search  areas  for  groups  of  UGVs.  Detected  threats  were 
neutralized  by  simply  clicking  on  the  threat’s  icon  in  the  control  station’s  interface.  Performance  was 
measured  by  the  number  of  hits  that  the  mobile  threats  inflicted  on  the  convoy  when  the  vehicles  were  within 
weapon  range. 

Figure  12-1  shows  an  example  of  interface.  The  pane  on  the  left  was  used  for  allocating  UGVs  into  colour 
coded  search  groups  and  selecting  groups  for  designation  of  search  areas.  Six  search  groups  were  available  in 
both  conditions,  whether  the  tactical  reconnaissance  was  performed  using  six  or  twelve  UGVs.  After  selecting 
a  search  group,  the  operator  drew  the  search  area  on  the  map  to  the  right.  The  map  could  be  scrolled  and 
zoomed  in  and  out  any  time  during  a  trial.  The  position  of  the  convoy  vehicles,  UGVs,  and  detected  threats 
where  indicated  on  the  map.  The  detection  of  threats  was  also  indicated  by  changing  the  icon  colour  of  the 
corresponding  UGV  in  the  left  pane  and  sounding  an  auditory  alarm.  Convoy  vehicles  that  were  recently  hit 
by  a  threat  were  also  indicated. 
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Figure  12-1:  Example  of  the  Operator  Control  Station.  The  blue  circles  indicate  convoy  vehicles.  The 
yellow  and  pink  circles  indicate  UGVs  in  colour  coded  search  groups.  The  red  circle  (encircled  in 
white  for  clarity)  in  the  upper-left  corner  of  the  map  in  the  industrial  area  shows  a  mobile  threat 
approaching  the  convoy.  The  UGVs  that  detect  the  threat  have  a  red  icon  in  the  left  pane. 
Convoy  vehicles  that  were  recently  hit  by  a  mobile  threat  are  shown  with  an  orange  colour. 


12.4  TECHNOLOGIES  EXPLORED 

A  partly  autonomous  search  function  for  multiple  UGVs  was  investigated  that  minimize  the  time  for  capture 
rather  than  guaranteeing  capture  [3].  The  algorithm  simplifies  the  search  problem  by  composing  the 
environment  into  convex  cells  where  an  UGV  that  enters  the  area  of  a  cell  can  always  detect  any  threats  that 
are  present  within  the  cell.  Initially,  there  is  an  equal  probability  that  there  is  a  threat  within  all  the  convex 
cells  within  a  search  area.  Over  time,  however,  these  probabilities  change  depending  on  the  connections 
between  adjacent  convex  cells  and  which  cells  the  UGVs  visit.  The  algorithm  prioritizes  which  adjacent 
convex  cell  to  visit  using  a  heuristic  entropy  function  that  minimizes  the  entropy  over  five  consecutive  cells 
starting  from  the  UGV’s  current  cell. 
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12.5  HUMAN  FACTORS  ISSUES  EXPLORED 

There  are  many  potential  applications  for  robotic  systems  that  are  only  feasible  if  one  or  a  few  operator  can 
simultaneously  control  several  robots.  One  way  to  enable  such  multi-robot  systems  is  to  use  partly  autonomous 
functions  where  the  robots  can  maintain  an  acceptable  performance  level  even  when  unattended  to  by  the 
operator.  However,  similarly  to  many  automated  systems,  partly  autonomous  functions  are  most  successful  in 
applications  that  only  require  limited  interaction  between  the  operator  and  the  autonomous  function  (e.g.  [4]). 
Applications  that  require  much  interaction  are  more  likely  to  have  human-robot  coordination  problems. 
An  important  factor  for  the  interaction  are  the  dependencies  between  the  task  that  operator  performs  and  the  task 
that  the  partly  autonomous  function  performs.  Tasks  with  only  weak  dependencies  are  more  likely  to  be  more 
suitable  for  multi-robot  systems  than  tasks  with  strong  dependencies  [5]. 

The  purpose  of  the  study  was  investigate  how  the  operators  perceived  the  dependencies  between  the 
designation  of  search  areas  and  the  partly  autonomous  search  function  in  a  representative  urban  environment. 
With  weak  task  dependencies,  an  increased  number  of  UGVs  for  tactical  reconnaissance  should  reduce  the 
number  of  hits  on  the  convoy  without  any  significant  effect  on  the  operator’s  task.  Weak  task  dependencies 
were  expected  since  search  tasks  generally  have  weaker  dependencies  than  other  tasks,  such  as  navigation  [1]. 
Please  see  Svenmarck  et  al.  [1]  for  more  information  about  the  study. 


12.6  UNMANNED  SYSTEMS  USED 

Six  or  twelve  simulated  UGVs. 


12.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

The  results  of  the  study  have  been  presented  at  Task  Group  meetings,  as  well  as  at  conference  sessions 
arranged  by  Task  Group  members  [1],[2].  The  following  table  summarizes  the  extent  of  the  NATO  collaboration. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

Coordination 

Collaboration 

12.8  SUMMARY  OF  TD  RESULTS 

The  results  show  that  increasing  the  number  of  UGVs  from  six  to  twelve  reduce  the  numbers  hits  on  the 
convoy  by  25%,  due  to  a  26%  increase  in  detected  threats  and  a  33%  increase  in  neutralized  threats. 
Subjectively,  the  operators  also  reported  higher  situation  awareness  as  measured  by  3D-SART.  More 
importantly,  these  benefits  were  achieved  without  any  detrimental  effects  on  the  mental  workload  as  measured 
by  NASA-TLX.  Overall,  the  operators  used  fairly  large  search  areas  and  a  uniform  allocation  of  the  number  of 
UGVs  into  search  groups. 
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12.9  LESSONS  LEARNED 

Typical  issues  in  control  of  multi-robot  systems  can  be  investigated  in  a  laboratory  environment  without 
experienced  operators,  although  care  should  be  taken  when  interpreting  the  results. 


12.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  operators  were  college  students  and  thus  rather  naive  towards  the  complexity  of  tactical  reconnaissance. 


12.11  CONCLUSIONS 

The  results  show  that  partly  autonomous  functions  can  improve  the  operator  to  vehicle  ratio  in  a  military 
environment  if  there  are  weak  dependencies  between  the  task  that  the  operator  performs  and  the  task  that 
partly  autonomous  function  performs.  However,  the  task  dependencies  also  partly  depend  on  the  operators’ 
task  experience,  as  well  as  the  formal  task  properties.  The  operators’  control  strategy  shows  that  they  were 
rather  naive  towards  the  complexities  of  tactical  reconnaissance.  More  experienced  operators  may  therefore 
find  different  task  dependencies. 


12.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

One  future  research  need  is  better  theories  for  conceptual  evaluation  of  task  dependencies  to  reduce  the  need 
for  empirical  investigations  when  introducing  partly  autonomous  functions.  Future  studies  may  also 
investigate  suitable  control  strategies  for  comparison  with  operator  performance,  as  well  as  to  support  the 
operators’  control  decisions.  Finally,  mixed-initiative  control  may  be  investigated  by  varying  the  operators’ 
degrees  of  freedom  in  designating  search  areas.  There  are  currently  no  plans  for  future  research,  however. 
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13.1  DATES 

This  chapter  concerns  the  series  of  technology  demonstrations  and  test  and  evaluation  trials  conducted  under  the 
United  Kingdom  (UK)  Ministry  Of  Defence  (MOD)  Dynamic  Airborne  Mission  Management  (DAMM) 
research  programme.  DAMM  is  a  set  of  principles,  interfaces  and  interactions  for  the  delivery  of  effects,  enabled 
by  advanced  digital  networking  and  mission  enabling  technologies,  providing  a  distributed,  collaborative  and 
adaptive  mission  capability  for  stability  and  dominance  in  a  dynamic  environment.  The  research  was  performed 
by  UK  Defence  Industry,  led  by  QinetiQ,  with  other  Industry  participants,  including  Thales,  BAES,  General 
Dynamics  and  AugustaAVestland.  The  research  was  performed  under  the  DAMM  Capability  Concept 
Demonstrator  (CCD)  research  programme,  with  underpinning  research  provided  under  the  MOD  Applied 
Research  Programme  (ARP),  Research  Entity  (RE)  314,  Mission  Enabling  Technologies  and  Demonstration 
(MET&D).  The  work  was  sponsored  by  MOD  Cap  TA  (Capability  -  Theatre  Airspace)  and  contracted  through 
DE&S  FBG-3  (Defence  Equipment  and  Support,  Future  Business  Group),  with  programme  support  technical 
advice  provided  by  MOD  Defence  Science  and  Technology  Laboratory,  Air  Weapons  and  Systems  Department 
(DSTL  AWSD).  International  Research  Collaboration  (IRC)  on  DAMM  was  conducted  between  MOD/Dstl  and 
United  States  (US)  Air  Force  Research  Laboratory  (AFRL),  711th  Human  Performance  Wing  (HPW),  Human 
Effectiveness  Directorate  (HEC),  Warfighter  Interface  Division,  under  the  auspices  of  the  US-UK  Project 
Arrangement  (PA),  Network  Centric  Strike  Controllers  (NCSC). 

The  UK  MOD  DAMM  CCD  programme  with  QinetiQ  commenced  on  contract  in  December  2008, 
with  completion  of  Phase  3  scheduled  for  July  2011.  The  underpinning  UK  MOD  research  programme  RE314 
MET&D  with  QinetiQ  provided  Mission  Management  System  (MMS)  technical  work  and  risk  reduction  for 
DAMM  CCD  throughout  this  period,  with  Synthetic  Environment  (SE)  trials  and  technology  demonstrations. 

The  DAMM  CCD  programme  was  conducted  over  3  years,  and  divided  into  three  phases,  as  illustrated  by  the 
Command  and  Control  -  Mission  Management  (C2-MM)  architecture  in  Figure  13-1.  Each  phase  advanced 
the  degree  of  DAMM  architecture  complexity  and  built  upon  the  previous  phase  to  demonstrate  enhanced 
capability: 

•  Phase  1,  YR1  (2008  -  2009)  Demonstration  of  baseline  DAMM  architecture  including  a  Fixed  Wing 
(FW)  Tactical  Fast-Jet  (TFJ)  package  co-ordinated  by  a  C2  element. 

•  Phase  2,  YR2  (2009  -  2010)  Integration  of  Rotary  Wing  (RW)  MMS  into  the  Phase  1  architecture  and 
demonstration  of  rapid  re -planning  between  assets  in  close  support  of  land  forces. 

•  Phase  3,  YR3  (2010  -  201 1)  Integration  and  demonstration  of  an  autonomous  Uninhabited  Air  System 
(UAS)  element  to  the  Phase  2  for  co-ordinated  targeting  capability. 
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Figure  13-1:  DAMM  C2-MM  Architecture. 


Under  RE314  MET&D  and  DAMM  CCD,  over  ten  technology  demonstrations,  SE  and  flight  trials  were 
completed  in  total;  the  four  activities  involving  UAS  components,  reported  here,  were  as  follows: 

•  RE314  MET&D  Joint  US-UK  SE  Trial,  September  2010  -  FW  TFJ  and  RW  AH/SH  Inter-Flight 
Co-ordination  with  FAC/JTAC,  C2  and  UAS. 

•  DAMM  CCD  UAS  SE  Technical  Demonstration,  April  2011  -  UAS  Co-ordination  with  RW  SH, 
FW  TFJ,  C2  and  FAC/JTAC. 

•  RE314  MET&D  Joint  US-UK  SE  Trial,  July  2011  -  Multiple  UAS  Co-ordination  with  RW  SH, 
FW  TFJ,  C2  and  FAC/JTAC. 

•  DAMM  CCD  RW  Flight  Trial,  August  201 1  -  RW  AH  and  RW  SH  Co-ordination  with  FW  TFJ,  C2, 
UAS  and  Deployed  FAC/JTAC. 


13.2  LOCATION 

RE314  MET&D  and  DAMM  CCD  were  UK  Ministry  of  Defence  research  programmes  managed  by  Dstl 
AWSD  from  Famborough  (2001  -  2009)  and  Portsdown  West  (2009  -  2011)  in  Hampshire  UK.  The  work 
was  performed  under  contract  with  QinetiQ  Mission  Management  Group,  based  at  Famborough,  Hampshire, 
UK.  Technology  development,  laboratory  studies  and  SE  trials  were  conducted  at  QinetiQ  Famborough;  flight 
trials  were  conducted  in  various  UK  airspace  geographic  locations. 
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13.3  SCENARIO/TASKS 

DAMM  comprised  the  following  tactical  mission  capabilities: 

•  Airborne  re-planning/modification  of  a  pre-planned  mission  in  response  to  dynamic  events; 

•  Airborne  plan  formulation  for  missions  which  cannot  be  planned  in  advance;  and 

•  Airborne  co-ordination  and  de-confliction  of  multiple  packages  with  a  mission  and  between  multiple 
missions. 

RE314  MET&D  and  DAMM  CCD  focused  on  air-to-ground  missions  with  increasing  emphasis  on  improved 
air-land  integration.  Each  phase  included  increased  integration  levels  with  relevant  C2  and  Intelligence 
Surveillance  Target  Acquisition  and  Reconnaissance  (ISTAR)  systems. 

•  Tactical  Fast  Jet  (TFJ)  Scenario  -  Unplanned  Time  Sensitive  Target  (TST)  and  Close  Air  Support 
(CAS)  mission  co-ordinated  with  Suppression  of  Air  Defence  (SEAD)  support;  C2  (Combined  Air 
Operations  Centre  (CAOC)  /  E3  AWACS)  to  TFJ  co-ordination  (Figure  13-2). 


Figure  13-2:  DAMM  TFJ  SE  Trial  NDZ  Scenario  Routes,  Threat  Locations  and  Decision  Points. 


•  Rotary  Wing  (RW)  Scenario  -  Stop  and  detainment  of  leadership  target,  co-ordinated  with  Joint 
Terminal  Area  Control  (JTAC)  /  Forward  Air  Control  (FAC)  and  TFJ  support;  C2  to  helicopter  MMS 
and  TFJ  co-ordination;  Joint  Personnel  Recovery  (JPR);  Dynamic  digital  reallocation  of  airspace. 

•  Uninhabited  Air  Systems  (UAS)  Scenario  -  Providing  ISTAR  capability  extension  of  the  RW 
scenario,  with  reconnaissance,  observation,  and  overwatch;  TFJ  CAS/TST  co-ordination. 

DAMM  development  used  a  validated  Joint  Warrior  military  scenario  based  on  a  NATO  Joint  Training 
Exercise  (Joint  Combat  Aircraft  /  Maritime  Integrated  Systems  Capability).  This  Joint  Warrior  scenario 
provided  military  realistic  assets  and  a  validated  Air  Tasking  Order  (ATO)  and  Airspace  Co-ordination  Order 
(ACO).  The  fictional  scenario  involved  geo-political  tension  and  the  early  phase  of  an  escalating  conflict 
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between  adjacent  national  interests  with  disputed  border  regions.  Neighbouring  Nation  states,  named  Avalon, 
Caledonia  and  Dragonia,  were  in  dispute  over  two  territorial  zones  ,  with  the  Northern  Disputed  Zone  (NDZ) 
set  for  convenience  in  Argyll  in  the  north-west  Scotland,  and  the  Southern  Disputed  Zone  (SDZ)  in  Wiltshire 
and  Somerset  in  south  west  of  England.  Scenario  missions  were  operations  of  a  multi-national  coalition  force 
assembled  to  maintain  stability  with  backing  by  United  Nations  Security  Council  Resolutions  (UNSCR). 
Coalition  forces  operated  against  insurgents,  threats  and  covert  activities  (e.g.,  weapons  importations  and 
movements),  with  intelligence  on  High  Value  Targets  (HVT).  Coalition  operations  involved  insertion  and 
extraction  of  forces  and  the  maintenance  of  logistical  support.  The  Coalition  had  air  superiority  within  the 
disputed  airspace.  Coalition  operations  had  defined  political  and  geographical  settings  (POL-MIL),  Areas  Of 
Responsibility  (AOR),  and  monitored  enemy  Order  of  Battle  (ORBAT),  including  re-locatable  air  defence 
units  threatening  operations  close  to  borders  and  within  disputed  territory.  Missions  were  briefed  with  Package 
composition,  Commanders  Intent  (e.g.,  increasing  stability  in  AOR)  and  Rules  Of  Engagement  (ROE), 
with  discrete  operational  phases  (e.g.,  seize  initiative,  dominate). 

In  planning  the  scenario  mission,  a  balance  was  sought  in  between  the  requirements  of  operational  realism  and 
the  need  to  exercise  and  practice  the  DAMM  tools.  Consideration  was  given  to  factors  influencing  the 
dissemination  of  command  intent,  operational  and  tactical  task  evaluation  and  task  allocation,  and  tactical 
mission  management.  Factors  assessed  included  objectives  to  accomplish,  threats  to  success,  environmental 
factors,  ROE,  and  agility.  This  analysis  was  used  to  elicit  prioritisation  of  key  mission  system  stressors  across 
aircraft  roles,  judged  as  likely  to  be  mitigated  by  tool  usage,  and  significantly  impacting  on  dynamic  mission 
effectiveness.  The  generic  mission  stressors  identified  and  employed  in  the  scenarios  were  as  follows: 

•  Electronic  Order  of  Battle  (EOB)  Change; 

•  Ability  to  Interoperate  (Communications); 

•  Force  Intent  Change; 

•  Force  Capability  Degradation; 

•  Weather;  and 

•  ROE  Change. 

Scenarios  and  tasks,  with  stressing  decision  points  (e.g.,  Figure  13-2),  employed  for  development  and  testing 
used  appropriately  realistic  operational  contexts,  with  representative  C2  system  architectures,  current  Concept 
of  Operations  (CONOPS)  and  Concept  of  Employment  (CONEMP),  Standard  Operating  Procedures  (SOPs), 
Special  Instructions  (SPINS)  and  ROE.  Qualified  and  experienced  serving  military  commanders  and  operators 
were  used  as  military  advisors  and  test  participants  to  ensure  best  use  of  current  military  procedures,  tactics 
and  planning  assumptions. 


13.4  TECHNOLOGIES  EXPLORED 

DAMM  CCD  integrated  mature  MMS  developed  under  the  RE314  MET&D  underpinning  research  programme. 
DAMM  MMS  comprised  Situation  Awareness  (SA)  tools,  collaboration  aids  and  decision  support  techniques, 
across  dissimilar  platform  types,  exploiting  and  advancing  research  conducted  within  FJ,  RW  and  UAS  domains. 
These  MMS  were  further  integrated  with  C2,  ISTAR  and  ground-force  elements  to  demonstrate  the  benefits  and 
effectiveness  of  near-term  airborne  Network-Enabled  Capability  (NEC)  using  current  data  link  technology. 
A  key  component  of  the  NEC  is  information  management,  including  maximizing  effective  use  of  available  radio 
frequency  spectrum  and  Human  Machine  Interface  (HMI)  which  supports  current  doctrine  and  operational 
methods. 
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The  CCD  demonstrated  and  evaluated  architectures  for  enabling  DAMM  at  the  tactical  level  including: 

•  Identification  of  information  management  issues  and  development  of  strategies  that  make  the  most  of 
the  effective  use  of  the  available  bandwidth;  and 

•  Information  Exchange  Requirement  (IER)  understanding/development  leading  to  definition  of  bandwidth 
efficient  protocols  and  Interface  Control  Definitions  (ICD). 

In  terms  of  Technology  Readiness  Levels  (TRL),  the  RE314  MET&D  research  programme  raised  relatively  low 
TRL,  early  concept  prototype  development  work  (TRL  1  -  3)  to  TRL  5  by  demonstration  in  a  representative  SE 
environment.  TRL  5  was  the  entry  level  for  MMS  technology  insertion  into  the  DAMM  CCD  programme  for 
flight  trial  demonstration  and  for  raising  to  TRL  7. 

The  DAMM  sub-system  spans  the  Operational  and  Tactical  C2-MM  layers.  DAMM  is  based  on  a  three  tier 
C2-MM  architecture  designed  to  provide  sensible  and  adaptive  spans  of  control  needed  for  tight  dynamic 
tactical  co-ordination.  The  three  tiers  differ  in  the  roles  and  functionality  provided,  and  the  kinds  of  decision 
support  tools  needed.  The  three  tiers  and  associated  tools  are  identified  as  follows: 

•  Tier  1  Operational  C2  Level  -  Ground  (e.g.,  CAOC)  or  airborne  (e.g.,  AW  ACS  E-3)  C2  nodes: 
OpTEAM  -  Operational  Task  Evaluation  and  Authorisation  Manager. 

•  Tier  2  Tactical  C2  Level  -  Mission  Commander,  Package  Commander:  TacTEAM  -  Tactical  Task 
Evaluation  and  Authorisation  Manager;  DESCAT  -  Digital  Exchange  System  for  Control  and  Targeting 
FAC  equipment. 

•  Tier  3  Effector  Levels  -  Flight  formation  members:  TDSS  -  Tactical  Decision  Support  System; 
OMPS  -  Rotorcraft  TDSS  On-board  Mission  Planning  System;  UAS  Ground  Control  Station  (GCS). 

OpTEAM,  TacTEAM  and  TDSS  comprised  different  sets  of  DAMM  MMS  networked  SA  tools,  collaboration 
aids  and  Decision  Support  System  (DSS)  techniques,  as  needed  for  supporting  tactical  co-ordination  and 
decision  making  at  the  three  tier  levels.  Examples  of  the  DAMM  tools  are  illustrated  in  Figure  13-3. 
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Dynamic  Air  Mission  Management 

To  assist  commander  &  aircrew  planning, 
decision  making  &  coordination. 

To  reduce  time  from  target  detection  to 
prosecution  for  complex  stressing  missions. 

To  be  achieved  by: 

■  Airborne  replanning/modification  of  a 
preplanned  mission  in  response  to  dynamic 
events 

•  Airborne  plan  formulation  for  missions  which 
cannot  be  pre-planned  in  detail 

■  Airborne  coordination  &  deconfliction  of 
multiple  packages  within  a  mission  &  between 
multiple  missions 

Dstl/AWSD  with  Q'nenQ 

MOD  Reseorcn  Sponsors:  CAP  TAScl  AR/DES  FBG-3d 


Images  courtesy  of  QinetiQ.  Farnborough ,  Airborne  Mission  Management 


Figure  13-3:  Examples  of  DAMM  Networked  SA  Tools,  Collaboration 
Aids  and  Decision  Support  System  Techniques. 

OpTEAM  provided  the  Tier  1  Operational  C2  layer  (CAOC/AWACS  E3)  with  the  following  tool  features 
(Figure  13-4): 

•  Map  display;  multiple  map  types;  zoom  options;  lat  long  read  out,  distance  measures  and  unit  conversion; 
cultural,  aeronautical  and  tactical  overlays;  multiple  display  options  (e.g.,  de-clutter);  inter-visibility 
analysis;  the  ability  to  create  and  review  mark-up. 

•  Datalink  capability  (real  and  simulated);  ATO  and  ACO  import  and  display  (along  with  other  IER 
data  feeds);  digital  Global  Area  Reference  System  (GARS)  airspace  de-confliction  and  management; 
chat  facility;  current  position  of  assets;  positions  of  threats,  targets,  forces,  and  wingmen. 

•  Routing  information  (for  multiple  assets  and  packages);  route  editing  capability;  route  de-confliction; 
route  timing  information;  route  rehearsal  and  three  dimensional  (3D)  fly  through. 

•  Aircraft  status;  predicted  fuel  levels  and  tanking  plans. 
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Figure  13-4:  OpTEAM  Supported  Airborne  Warning  and  Control  SE  Flying  Desk. 

Under  DAMM  CCD,  Thales  Air  Systems  Division  provided  an  advanced  web-enabled  solution  for  OpTEAM 
C2  based  on  their  WebS2AT  product  with  ATO/ACO  visualisation,  supplemented  by  the  Recognised  Air 
Picture  (RAP)  and  feeds  from  the  Joint  Operation  Picture  (JOP). 

The  following  tool  features  were  provided  by  TacTEAM  at  the  DAMM  Tier  2  Tactical  C2  layer,  and  by  TDSS/ 
OMPS  at  the  DAMM  Tier  3  effectors  layer  (TFJ,  RW)  (Figure  13-5): 

•  Moving  map  display;  multiple  map  types;  zoom  options;  cultural,  aeronautical  and  tactical  overlays; 
multiple  display  options  (e.g.,  brighten/dim  screen,  or  show  hide  wingmen  routings);  the  ability  to 
review  mark-up;  inter-visibility  analysis. 

•  Datalink  capability  (real  and  simulated);  digital  tasking;  Planned  Position  Location  Indication  (PPLI); 
positions  of  threats,  targets,  forces,  and  wingmen. 

•  Routing  information  (for  multiple  assets  and  packages);  route  editing  capability;  route  timing 
information. 

•  Aircraft  status;  chat  facility;  tactical  decision  support;  Collateral  Damage  Estimation  (CDE)  visualisation 
and  assessment. 
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Figure  13-5:  TacTEAM  Supported  TFJ  Weapons  System  Officer  SE  Flying  Desk. 


DESCAT  2  (Digital  Exchange  System  for  Control  and  Targeting)  was  provided  to  support  the  Forward  Air 
Controller  (FAC)  or  Joint  Terminal  Attack  Controller  (JTAC),  operating  at  DAMM  Tier  2  Tactical  C2  layer 
(Figure  13-6).  DESCAT  2  facilitated  digital  communications  between  ground  and  air  assets,  specifically  the 
designation  and  allocation  of  targets  to  assets  via  the  9-line  brief.  The  system  capabilities  and  Windows 
XP-based  user  interface  provided  the  following  tool  features  and  functions: 

•  Raster  map  display;  Digital  Terrain  Elevation  Database  (DTED)  terrain  database  height  information; 
multiple  map  types;  zoom  options;  multiple  display  options  (e.g.,  brighten/dim  screen,  de-clutter); 
lat  long  read  out,  distance  measures  and  unit  conversion;  tactical  overlays;  receipt  and  display  of 
imagery,  representative  of  UAS  ROVER  capability;  ability  to  create  and  review  mark-up;  route 
forwarding;  Collateral  Damage  Estimation  (CDE)  visualisation  and  assessment. 

•  Improved  Data  Modem  (IDM)  data-link  protocols  (TACFIRE,  AFAPD,  VMF),  real  and  simulated; 
digital  9  line  tasking;  receipt,  display  and  transmission  of  PPLI;  positions  of  threats,  targets,  Blue  forces; 
aircraft  status,  e.g.,  altitude;  indirect  target  mensuration  with  GPS  and  Laser  Rangefinders  interfaces. 

•  C2  control  interfaces  with  Joint  Automated  Deep  Operations  Co-ordination  System  (JADOCS) 
and  Cursor  On  Target  (COT)  XML  message  set;  local  control,  receipt,  display  and  transmission  of 
digital  Global  Area  Reference  System  (GARS)  airspace  de-confliction  and  management  messages; 
chat  facility. 
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Figure  13-6:  DESCAT  2  Supported  FAC/JTAC. 

Under  the  auspices  of  the  US-UK  NCSC  /  Strike  Warrior  II  PA  IRC  programme,  USAF  AFRL  provided  the 
JTAC/FAC  Battlefield  Air  Targeting  Man  Aided  kNowledge  (BATMAN)  system  for  DAMM  SE  integration 
and  testing  with  DESCAT  2.  The  BATMAN  Bareback  software  application  provided  an  electronic  9  line 
capability,  with  target  data  input  via  Laser  Range  Finder  (LRF)  or  manually,  using  Speech  Software  with 
voice  feedback,  and  friendly  data  input  via  Global  Positioning  System  (GPS)  or  manually,  with  digital  data 
transmission  to  CAOC.  The  BATMAN  system  provided  operator  aiding  via  Falconview  mapping  and  terrain 
visualisation,  with  decision  support  information  for  the  Special  Tactics  (ST)  operator,  and  UAV  tools. 
The  system  included  a  3D  audio  and  DRAW  mark-up  research  capability. 

UAS  GCS,  based  on  UK  Watchkeeper  (WK)  Tactical  Unmanned  Air  Vehicle  (TUAV),  provided  the 
following  functions  and  capabilities  at  DAMM  Tier  3  Effectors  layer  (Figure  13-7): 

•  Moving  map  display;  multiple  map  types;  zoom  options;  cultural,  aeronautical  and  tactical  overlays; 
routing  information  (for  multiple  assets  and  packages);  route  editing,  timing  information,  and  route 
following  capability;  inter-visibility  analysis;  mark-up;  multiple  display  options  (e.g.,  brighten/dim 
screen,  co-operating  platforms  routes);  route  assessment  tools,  e.g.,  threat  envelopes,  airspace  constraints; 
target  mensuration. 
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•  Datalink  capability  (real  and  simulated);  digital  tasking;  Planned  Position  Location  Indication  (PPLI); 
positions  of  threats,  targets,  forces,  co-operating  platforms;  aircraft  status. 

•  Real-time  sensor  video;  sensor  footprint  overlay;  vehicle  controls,  e.g.,  camera  guide  mode,  auto-track 
mode. 


Figure  13-7:  UAS  GCS  Based  on  WK  TUAV. 

Current  UAS  operate  as  single  platforms  under  Remotely  Piloted  Vehicle  (RPV)  control.  With  more 
autonomous  UAS  systems,  it  is  likely  that  the  operator  will  be  controlling  multiple  UAVs,  using  for  example  a 
goal-oriented,  service  request-based  approach.  A  multi-UAS  GCS  operator  could  be  considered  as  operating 
more  at  the  DAMM  Tier  2  Tactical  C2  layer. 

A  C2-MM  capability  scale  was  used  to  summarise  the  technologies  involved  and  the  impact  of  capability 
enhancement,  as  shown  in  Table  13-1  below.  This  capability  maturity  scale  provides  levels  coupling  network 
communications  and  collaborative  decision  support  technologies  and  interfaces  for  C2-MM.  The  DAMM 
programme  spanned  Level  2  (DAMM  Baseline  architecture)  to  Level  4/5  (DAMM  Objective  architecture). 
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Table  13-1 :  Levels  of  Capability  Maturity  for  C2-MM. 


C2-MM 

Capability 

Level 

Communications 

Mission  Management  Decision  Support 

1 

Voice  Radio 

No  onboard  integrated  planning  tool  -  Aircrew 
unsupported  (re-planning  and  reactive) 

2 

Legacy  Data  Link  (Limited 
message  set) 

Limited  or  no  onboard  planning  aid,  limited  or  no 
onboard  collaboration  tools,  co-ordinated  planning 
and  co-ordination  by  voice 

3 

Legacy  Data  Link  (Enhanced 
Message  Set  (EMS)) 

Intra-flight  MM,  adaptable  decision  support, 
co-ordinated  machine-machine  planning  within 
package 

4 

Legacy  Data  Link  (EMS) 

Inter- flight  MM  within  and  across  missions, 
adaptable  decision  support,  fully  integrated  with 
operational  C2  (year  1  DAMM) 

5 

Legacy  Data  Link  (EMS) 
5th"Generation  Capability 

Full  adaptive/adaptable  MM,  dynamic  battle 
management  supported  by  (limited  by)  legacy  data 
link.  Cross-domain  (air  and  land)  dynamic 
co-ordinated  planning. 

6 

Advanced  Data  Link 

As  above,  but  high  capacity  ad  hoc  network, 
agility,  scale,  richness 

7 

Advanced  Data  Link 

As  above,  with  pro-active  Decision  Support 

System 

13.5  HUMAN  FACTORS  ISSUES  EXPLORED 

DAMM  enables  the  provision  of  capability  that  is  efficient  and  effective  in  a  complex  dynamic  tactical 
environment.  Complex  high  tempo  operations,  such  as  TST,  CAS,  SEAD  and  JPR,  require  persistence  and 
urgency,  and  need  rapid  co-ordination  and  collaboration,  both  within  and  between  mission  packages, 
to  deliver  complex  effects  and  ensure  mission  success.  Critically,  a  DAMM  collaborative  capability  is  needed 
to  provide  operational  flexibility,  versatility  and  adaptability,  enabling  responsivity  to  rapidly  changing 
mission  requirements  and  dynamic  events,  such  as  re-tasking  or  threat  changes.  A  collaborative  and  adaptive 
DAMM  capability  requires  important  Mission  Essential  Competencies  (MEC),  involving  interpretation  of 
command  intent,  rapid  situation  assessment,  re-planning  and  communication,  all  with  decision  making  for 
decision  superiority  at  the  core.  Thus,  the  main  Human  Factors  (HF)  research  thrust  under  DAMM  was 
improving  Critical  Mission  Decision  Making  (CMDM)  in  a  distributed,  collaborative,  adaptive  environment. 

The  HF  issues  of  primary  interest  concerned  understanding  the  requirements  for  aircrew  interactions  and 
interfaces  needed  for  improving  collaborative  CMDM,  and  the  development  of  applicable  analytical  tools  and 
techniques  for  DAMM  requirements  capture,  design,  development,  testing  and  evaluation.  The  focus  was  on  the 
Operator-Mission  Interface  (OMI),  and  on  the  mission  system  functional  and  cognitive  requirements,  rather  than 
on  HMI  ergonomics.  Relevant  OMI  issues  included  the  following:  mission  plan  co-ordination  and  critiquing; 
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the  centralisation,  distribution  and  hierarchical  structure  of  mission  command  flows;  the  communication  and 
interpretation  of  command  intent;  prioritisation  of  mission  critical  decision  information. 

Specifically,  the  HF  issues  explored  under  DAMM  were  concerned  with  the  following:  developing  HF 
methods  for  capturing  user  functional  and  information  requirements  for  OMI  CMDM  in  a  distributed  networked 
environment  using  SE  technical  demonstration  testing  and  flight  trials;  modelling  and  measurement  of  the 
effects  of  highly  networked  MMS  mission  enabling  technologies  OMI,  and  associated  collaboration  and 
decision  support  tools,  on  CMDM;  understanding  and  supporting  CMDM  processes  at  the  tactical  level, 
with  cross-capability  applicability,  and  within  the  broader  system-of-system  and  C2  context;  understanding 
the  impact  of  C2-MM  technologies  on  Mission/Tactical  Director  (TD),  Mission  Commander  (MC)  and 
Package  Commander  (PC)  roles  and  responsibilities,  and  Mission  Essential  Competencies  (MEC). 

Requirements  for  CMDM  processes  involved  in  DAMM  task  MECs  needed  investigation,  such  as  the  following 
9  CAS/TST  examples: 

1)  Formulation,  approval  and  dissemination  of  9  Line; 

2)  Task  allocation; 

3)  Weapon  target  matching; 

4)  EOB  update; 

5)  Dissemination  of  EOB  change; 

6)  Re -plan  of  tactical  level  assets; 

7)  Battle  Damage  Assessment  (BDA)  /  Battle  Damage  Identification  (BDI); 

8)  Re-attack  decision;  and 

9)  Accept  re-strike. 

Throughout  the  DAMM  demonstration  programme,  trial  planning  and  analysis  prioritised  the  identification  of 
scenarios  and  vignettes  with  operationally  significant  CMDM  events  and  triggers.  This  was  to  provide  valid 
and  relevant  use  cases  for  testing  the  DAMM  CMDM  tools  in  a  realistic  operational  context,  with  current 
SOPs,  ROE,  CONOPS,  and  C2  system  architectures. 

So,  the  HF  challenge  was  to  develop  and  exploit  HF  frameworks  and  protocols  appropriate  for  highly 
networked,  distributed,  collaborative,  adaptive  DAMM  CMDM  assessment,  and  to  support  modelling  and 
measurement  of  DAMM  CMDM  to  improve  utility  and  scientific  robustness,  i.e.,  improved  validity  and 
reliability,  sensitivity  and  discrimination,  diagnostic  and  prognostic  power. 

Development  of  DAMM  capability,  and  development  of  the  DAMM  CMDM  assessment  approach,  needed  to 
be  sensitive  to  the  following:  mission  context,  particularly  the  changing  conditions  on  the  ground;  command 
intent  and  ROE;  exchange  of  relevant  SA  information. 

DAMM  capability  development  needed  to  deliver  timely  Network  Enabled  Effects  in  response  to  dynamic 
events,  such  as  the  following:  reducing  time  from  task  nomination-to-effect  delivery;  reducing  the  time  from 
target  detection  to  prosecution;  co-ordinated  and  planned  Time  On  Target  (TOT  exchange);  accuracy  and 
precision  in  CDE  and  BDA/BDI. 

Development  of  the  DAMM  CMDM  assessment  approach  needed  to  improve  requirements  capture  and 
analysis,  and  improve  assessing  the  effectiveness  of  collaborative  decision  support  tools.  Applicable  methods 
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considered  included:  models  and  metrics  for  individual,  team  and  organisational  decision  making; 
C2  Measures  of  Performance  (MoP);  C2  Measures  of  Effectiveness  (MoE);  system-of-systems  views  and 
system  architecture  representations  of  decision  making  derived  from  MODAF/DODAF  system  architect  tools. 

The  strategy  for  DAMM  CMDM  assessment  development  was  to  extend  traditional  HF  metrics  of  speed  and 
accuracy,  SA  and  workload,  and  to  develop  models  and  metrics  of  decision  quality,  teamwork  and  C2-MM 
system  effectiveness,  such  as  dynamic  efficiency. 


13.6  UNMANNED  SYSTEMS  USED 

The  DAMM  research  programme  informed  understanding  of  the  requirements  for  current  and  future 
autonomous  UAS  to  support  DAMM  capability.  In  particular,  the  work  sought  to  understand  the  implications 
of  UAS  for  the  DAMM  architecture,  CONOPS  and  SOPs.  Integration  of  UAS  requirements  into  the  DAMM 
architecture  commenced  in  September  2010  under  Phase  3  of  the  programme.  In  Phase  3,  a  UAS  capability 
and  GCS  was  integrated  into  SE  work,  and  associated  MMS  into  the  DAMM  architecture,  based  on  the  UK 
WK  TUAV. 

Thales  is  the  primary  UK  supplier  of  UAS  capability  in-theatre.  Currently,  this  is  in  the  form  of  standard  Hermes 
450  air  platform  under  the  Lydian  programme  working  closely  with  Sea  King  ASaC.  This  is  coupled  in-theatre 
with  RAF  operated,  General  Dynamics  supplied,  Predator/Reaper  air  platforms.  Hermes  450  is  soon  to  be 
replaced  by  the  Thales  WK  system.  Thus,  WK  will  become  a  significant  element  of  the  UK’s  ability  to  provide 
ISTAR  in-theatre.  Currently,  the  contracted  WK  system  includes  an  Improved  Data  Modem  (IDM)  data  link, 
and  ROVER  video  down-link.  Building  on  this  WK  sensor  and  data  link  capability,  the  DAMM  system  sought 
to  provide  further  digital  networked  co-ordination  between  WK  and  C2  assets  (E3  AWACS),  other  tactical  air 
platforms  (RW  Support  Helicopter  (SH)  and  Attack  Helicopter  (AH);  FW  TFJ  strike  aircraft),  and  FAC/JTAC 
air-land  integration  assets.  WK  with  DAMM  enabled  capability  has  the  potential  to  provide  in-theatre  real-time 
ISTAR,  including  cross-cueing  of  one  asset  from  another  using  DAMM  digital  messages,  enabling  rapid  and 
accurate  dissemination  of  sensor  tasking  between  assets,  shortening  reaction  times,  enabling  more  contacts  to  be 
investigated,  and  effects  to  be  delivered  more  rapidly,  with  potentially  life  saving  consequences.  The  work 
demonstrated  co-ordinated  mission  planning  and  execution  between  manned  and  unmanned  platforms  as  part  of 
complex  missions.  The  simulated  WK  TUAV  with  IDM  data  link  provided  ISTAR  capability  including  the 
cross-cueing  of  one  asset  from  another,  using  digital  messages,  and  investigation  of  the  utility  of  direct  Full 
Motion  Video  (FMV)  feeds  from  WK  with  ROVER  video  down-link  (Figure  13-8).  This  final  phase  of  the 
DAMM  programme  continued  to  progress  TFJ  to  RW  integration,  as  well  as  air-land  integration  through 
FAC/JTAC  capability.  Op-TEAM,  Tac-TEAM  and  platform  MMS  incorporated  additional  tactics  required  to 
conduct  Phase  3  missions  with  UAV  elements. 
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C2  E3/AWACS  OpTEAM 


UAV  GCS 


OpTEAM  shows  DEVIL  68  Lynx  SH  helicopter  operating  on  route  between  two  sets  of  GARS  boxes 
Observer  UAV  PPLI  is  shown  operating  within  northern  pair  of  GARS  boxes.  Threat  is  update  ready  for  export. 
UAV  provides  virtual  RVT  BDI  of  2  Technicals  post  strike,  with  ID  of  location  of  3rd  target  vehicle  for  FJ  re-strike 

Figure  13-8:  C2  Co-ordination  of  RW  Support  Helicopter,  UAV  and  Fast  Jet. 


In  parallel  with  WK  TUAV  GCS  integration,  under  the  auspices  of  the  US-UK  NCSC  /  Strike  Warrior  II  PA 
International  Research  Collaboration  (IRC)  programme,  USAF  AFRL  provided  the  Multi -UAV  Supervisory 
Control  Interface  Technology  (MUSCIT)  Vigilant  Spirit  GCS  and  UAS  system  for  integration  and  testing  in 
the  DAMM  SE  environment.  This  was  used  for  investigation  of  advanced  UAS  capabilities  and  concepts  in 
the  RE314  MET&D  UAS  Integration  SE  Trial,  July  2011  (Figure  13-9).  MUSCIT  /  Vigilant  Spirit  is  a  single 
operator/multiple  TUAV  (x4)  system,  providing  UAS  ISTAR  capability  using  a  goal-oriented,  service 
request-based  approach  to  the  provision  of  imagery.  The  MUSCIT  /  Vigilant  Spirit  GCS  delivers  UAS 
imagery  on-demand  through  real-time  sensor  feeds  to  individual  DAMM  assets.  Individual  TUAVs  providing 
the  sensor  feeds  are  managed  by  the  GCS,  through  routing  optimisation  processes  that  are  transparent  to  the 
originators  of  service  requests.  By  introducing  this  supervisory  control  approach  for  a  multiple  TUAV  system, 
compared  with  UK  WK  TUAV  GCS,  the  integration  of  the  USAF  AFRL  MUSCIT  /  Vigilant  Spirit  UAS  GCS 
through  IRC  enabled  the  DAMM  research  programme  to  investigate  the  effects  of  UAS  GCS  working  more  at 
the  DAMM  Tier  2  Tactical  C2  layer. 
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Figure  13-9:  USAF  AFRL  MUSCIT  /  Vigilant  Spirit  TUAV  GCS  in  DAMM  SE. 

The  focus  of  the  DAMM  programme  was  on  de-risking  near-term  UAS  exploitation.  In  a  complementary 
research  activity,  briefed  and  demonstrated  to  NATO  HFM-170,  MOD/Dstl  sponsored  a  joint  Industry 
advanced  UAS  research  programme  with  QinetiQ,  BAES  and  Thales  entitled  Autonomy  and  Mission 
Management  (A&MM).  The  A&MM  research  programme  investigated  advanced  UAS  concepts  for  airspace 
and  mission  management  with  a  focus  on  relatively  low  TRL  development.  The  A&MM  programme  covered 
integration  of  tactical  UAV  (Thales  WK  TUAV)  and  operational  UAV  (BAES  MANTIS  OUAV)  capabilities, 
and  included  the  QinetiQ  Task  Execution  Framework  (TEF)  for  increasing  UAV  autonomy  of  heterogonous 
UAS  ISTAR  assets.  The  TEF  approach  uses  goal-based  tasking,  “person-to-purpose”  HMI,  and  agent  oriented 
software-based  planning  solutions  for  optimised  task  allocation,  co-ordination  and  routing.  In  this  A&MM 
work,  a  Service-Oriented  approach  to  the  provision  of  UAS  services  is  investigated,  moving  away  from 
current  process  of  tasking  specific  sensors/weapons  on  specific  platforms  to  collect  intelligence  or  deliver 
effects,  and  moving  towards  a  more  ‘Internet  Protocol  cloud-based  model’.  A&MM  programme  aspirations 
include  the  integration  of  autonomous  software  on  the  European  Common  Operating  System  (ECOS) 
architecture,  and  the  development  of  a  common  ASAC  three-layer  stack  architecture  (Hardware,  Common 
Operating  System  and  Applications  layers)  for  UAS  control. 
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13.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

The  DAMM  research  programme  focuses  on  UK  specific  requirements.  Reports  on  DAMM  status  and 
progress  were  routinely  communicated  to  NATO  through  HFM-170  meetings.  Involvement  of  NATO  Nations 
occurred  at  a  number  of  levels. 

US  AFRL  -  Direct  US  involvement  under  RE314  underpinning  SE  work  has  been  achieved  through  bilateral 
US-UK  NCSC  /  Strike  Warrior  II  PA,  between  MOD  Dstl  and  US  AFRL  71/HPW/HEC/Warfighter  Interface 
Division,  based  at  Wright  Patterson  AFB,  Dayton  Ohio.  This  provided  integration  of  AFRL  capabilities  into 
three  RE314  MET&D  US-UK  Strike  Warrior  II  SE  Trials,  including  Multi-Modal  Communications  (MMC), 
3D  spatial  audio,  US  JTAC  Batman  capability  imagery  mark-up  enhancements,  and  UAS  MUSCIT  Vigilant 
Spirit  GCS. 

Under  HFM-170,  discussions  on  DAMM  UAS  planning,  concerning  supervisory  control  and  HF  assessment 
issues,  have  been  held  with  AFRL  711/HPW/RHCI  (Dr.  Mike  Patzig  /  Dr.  Mark  Draper)  on  the  MUSCIT 
programme  and  Vigilant  Spirit  GCS.  This  was  supported  by  the  MUSCIT  multi-UAV  (x4)  flight  demonstration 
at  Camp  Atterbury,  Indiana  on  5  May  2010. 

US-UK  PA  collaboration  facilitated  by  HFM-170  discussions,  led  to  successful  integration  of  MUSCIT  / 
Vigilant  Spirit  GCS  capability  into  the  RE314  MET&D  3rd  US-UK  Strike  Warrior  II  UAS  Integration  SE 
Trial,  July  2011.  US-UK  IRC  enabled  the  DAMM  research  programme  to  investigate  the  effects  of  UAS  GCS 
supervisory  control  working  at  the  DAMM  Tier  2  Tactical  C2  layer. 

A  new  PA  on  UAS,  entitled  “Monitoring  and  Controlling  Multiple  Assets  within  Complex  Environments” 
(MC-MACE),  has  been  developed  between  AFRL  71 1/HPW/RHCI  and  MOD  Dstl,  lead  by  Brian  Donnelly  / 
Dr.  Mark  Draper  (US)  and  Robert  Taylor  /  Antony  Grabham  (UK).  The  MC-MACE  PA  is  a  multi-Nation 
programme  of  co-ordination  under  The  Technical  Co-operation  Programme  (TTCP),  Human  Resources  and 
Performance  (HUM)  Group,  Technical  Panel  7,  Human  Systems  Integration  -  Air.  This  TTCP  HUM  TP7  PA 
will  provide  increased  international  research  collaboration  on  UAS  programmes  involving  DRDC  Canada  and 
DSTO  Air  Operations  Division,  Australia,  in  addition  to  US  and  UK.  The  PA  will  support  new  UK  work  on 
Autonomy  and  Mission  Systems,  focussing  on  manned-unmanned  teaming  issues,  including  autonomy  and 
mission  management,  collaborative  autonomy  and  cohesive  collaborative  control  capability. 

US  Army  /NASA  -  US  Army  /  NASA  (Jay  Shively)  provided  advice  on  RW  scenarios,  Playbook  and  UAS 
issues  for  the  RE314  US-UK  Strike  Warrior  II  SE  Trials,  at  a  planning  meeting  held  in  San  Francisco,  USA  in 
May  2009. 

Sweden  FOI  -  Discussions  with  Sweden  FOI  FLSC  and  UK  MOD/Dstl  under  the  UK-SWE  MOU,  and 
supported  by  HFM-170,  have  led  to  planning  of  a  joint  UK-SWE  project  (Project  CODE)  on  distributed 
mission  simulation  and  synthetic  training  with  Live  Virtual  Constructive  (LVC)  capability.  A  meeting 
between  MOD  Dstl  (Bob  Taylor  /  Ebb  Smith  /  Robert  Anderson)  and  FOI  FLSC  (Jonathan  Borgvall  / 
Lars  Kristensson  /  Martin  Castner)  was  held  at  FLSC  on  3-4  March  2010  to  develop  scenario  vignettes. 
The  scenario  will  involve  network  linking  of  SE  facilities  at  FLSC  Stockholm  with  the  UK  MOD  Air  Battle 
Training  Centre  (ABTC)  at  RAF  Waddington.  The  first  UK-SWE  Exercise  SNOWSTORM  is  planned  for 
June  2011.  SNOWSTORM  will  focus  on  synthetic  training  requirements  for  operational  DAMM  capability. 

France  DGA  /  Telecom  Bretagne  -  Arising  from  meetings  under  HFM-170,  bi-lateral  meetings  between  UK 
and  France  on  DAMM  UAS  capability  and  HF  research  requirements,  have  been  held  in  UK  (4  December 
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2008,  Farnborough)  and  France  (1-2  April  2009,  Brest)  between  Institut  Telecom  /  Telecom  Bretagne 
(Dr.  Gilles  Coppin)  and  France  MOD/DGA  (Mr.  Didier  Bazalgette)  and  UK  MOD/Dstl  (Robert  Taylor  / 
Antony  Grabham).  In  parallel,  a  joint  Anglo-French  research  programme  on  Autonomy  and  Mission 
Management  has  been  agreed  commencing  2010.  The  aim  of  this  collaboration  is  to  ultimately  develop  a  joint 
SE  capable  of  allowing  dissimilar  UK  and  French  autonomous  UAS  to  interoperate  within  the  simulated 
battlespace  to  carry  out  multiple  missions  controlled  by  a  limited  number  of  operators. 

Australia  DSTO  AOD  -  Under  the  auspices  of  TTCP  (HUM  TP7/TP2),  DSTO  Australia,  Air  Operations 
Division  (POC  Dr.  Chris  Best)  had  direct  involvement  in  the  RE314  MET&D  2nd  US-UK  Strike  Warrior  II  SE 
Trial,  September  2010,  working  with  MOD  Dstl  and  US  AFRL  71 1/HPW. 

Understanding  of  the  implications  for  coalition  and  NATO  joint  operational  requirements  are  potentially  of 
interest  for  future  work.  The  main  area  for  NATO  involvement  was  in  the  planning  and  design  stages  of  the 
demonstration  activities,  with  possibilities  for  involvement  in  analysis,  as  summarised  in  Table  13-2.  There 
was  potential  for  collaboration  with  NATO  Nation’s  related  efforts  at  the  level  of  aims,  objectives,  approach, 
CONOPS,  metrics,  architectures,  and  in  particular  the  application  of  HF  lessons  learned. 

Table  13-2:  NATO  Collaboration  for  DAMM  CCD. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

X 

X 

Co-ordination 

X 

X 

Collaboration 

X 

X 

In  practice,  NATO  Nation’s  direct  involvement  in  the  execution  of  DAMM  activities  was  difficult  to  achieve, 
due  to  the  constraints,  funding  arrangements  and  complex  nature  of  the  project,  and  the  UK  specific 
requirements.  Nevertheless,  the  DAMM  network  interfaces  were  considered  ‘open’,  and  as  such,  NATO 
Nations  could  bring  elements  for  integration  into  future  demonstration  activities.  Collaboration  through  an  SE 
trial  is  probably  more  readily  achievable  than  through  a  joint  flight  trial. 


13.8  SUMMARY  OF  DAMM  UAS  TECHNOLOGY  DEMONSTRATION  RESULTS 

13.8.1  RE314  MET&D  Joint  US-UK  SE  Trial,  September  2010  -  FW  TFJ  and  RW  AH/SH 

Inter-Flight  Co-ordination  with  FAC/JTAC,  C2  and  UAS 

A  2nd  in  a  series  of  Joint  DAMM  SE  Trials  was  conducted  with  USAF  AFRL  at  QinetiQ  Farnborough  in 
September  2010,  under  the  US-UK  NCSC  /  Strike  Warrior  II  PA,  with  the  aim  of  increasing  Air-Land 
integration,  and  providing  further  development  of  collaboration  performance  measurement.  This  trial 
introduced  a  nominal  synthetic  UAS  component  for  future  capability  development.  The  trial  involved  USAF 
aircrew  and  UK  MOD  serving  military  operators,  with  additional  military  participation  provided  by  DSTO 
AOD  Australia,  under  TTCP  collaboration.  The  scenario  was  based  on  the  previous  FW/RW/C2  SE  Trial, 
under  Joint  Warrior  conditions,  with  increased  FAC/JTAC  involvement,  and  a  nominal  UAS  reconnaissance 
capability.  The  expanded  scenario  comprised  the  following  elements: 
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•  1  AWACS  E3-D  Airborne  C2  with  OpTEAM; 

•  2/3  FAC/JTAC  with  digital  targeting  technology  (DESCAT  and/or  BATMAN); 

•  2  FW  TFJ  Ground  Attack  aircraft  with  TacTEAM/TDSS  -  one  single  seat  aircraft  emulating  Typhoon 
capability  (FWSS),  and  one  twin  seat  aircraft  emulating  Tornado  GR4  (FW2S); 

•  2  RW  aircraft  -  one  RW  AH  and  one  RW  SH  -  both  operated  by  two  aircrew  with  standard  avionics 
(Baseline),  stand-alone  DIGIMAP  (Threshold)  or  networked  OMPS  Technology  (Objective);  and 

•  1  Reconnaissance  UAS  -  Platform  not  represented  but  simulated  sensor  feeds  provided  via  network  to 
different  nodes  depending  on  Baseline,  Threshold  or  Objective. 

White  Force  Trial  Control  was  provided  with  networked  OpTEAM  as  SA  and  trials  management  tool. 
Functionality  and  networking  varied  depending  on  Baseline,  Threshold  or  Objective.  The  trial  aim  was  to 
examine  the  effects  of  C2-MM  architectures  and  MC  positions  on  crew  decision  making.  Based  on  the  Joint 
Warrior  NDZ  scenario,  the  missions  involved  a  series  of  unexpected  events  requiring  re -planning  and  mission 
critical  decisions.  These  provided  discrete  decision  making  points  for  CMDM  assessment.  The  geographic 
location  was  centred  on  the  Invergarry  /  Fort  Augustus  /  Loch  Lochy  area  north-east  of  Fort  William  and  Ben 
Nevis.  The  scenario  involved  co-ordination  between  land  component  and  air  component  commanders 
prosecuting  CAS  missions  and  Time  TST  within  a  coalition  context  where  coalition  partners  are  undertaking 
pre-planned  missions.  Due  to  dynamic  events,  cross-co-ordination  between  the  CAS/TST  missions  and  the 
pre-planned  missions  that  were  originally  under  a  separate  AOR  becomes  necessary.  The  trial  sought  to 
improve  understanding  of  the  effects  on  mission  command  flow  and  locus  of  critical  mission  decision  making, 
such  as  the  balance  of  centralized  versus  distributed  decision  making. 

The  trial  followed  an  experimental  plan  designed  to  test  and  compare  three  alternative  C2-MM  architectures  - 
Baseline,  Threshold  and  Objective  -  and  three  positions  for  Mission  Commander  (MC)  -  C2  ,  RW  (AH), 
FW  (SS  and  2S)  -  with  nine  combinations  of  C2-MM  architectures  and  MC  positions  operating  over  three 
days,  with  three  different  runs  per  day.  The  scenario  was  controlled  by  White  Force  and  developed 
progressively  during  the  trial  runs  in  phases  using  three  mission  vignettes.  Different  events,  threats  and  target 
behaviours  were  produced  by  WF,  designed  to  maintain  operator  engagement  and  mitigate  learning. 
Performance  with  the  Baseline  architecture  was  tested  on  the  first  trial  run.  The  comparison  architectures  were 
tested  on  subsequent  runs  (Threshold  x  3:  Objective  x  4),  with  the  order  balanced  to  control  for  learning. 

DSTL  SMEs  provided  assessments  of  performance  and  effectiveness  based  on  identified  CMDM  critical 
decision  points.  These  were  obtained  during  post-run  de -briefing  sessions  from  participant  self-ratings  and 
SME  observer  scoring.  Subjective  ratings  (7-point  Low-High  anchors,  Likert  scales)  were  obtained  for  66  trial 
parameters  in  total.  These  comprised  21  participant  CMDM  self-ratings  for  identified  critical  decisions 
including  REMDAER  activity  phase  demand,  4T’s  change  management  demand,  workload,  SA  and  decision 
quality.  For  individual  trial  runs,  participant  ratings  were  obtained  for  tools  usability  (xl9),  and  for  generic 
system  CONOPS  and  architecture  assessment  parameters  (xl2),  including  dynamic  mission  efficiency, 
teamwork  and  technical  system  performance.  Additionally,  ratings  were  obtained  from  SME  observer  on  14 
assessment  scales  covering  workload,  decision  making,  dynamic  efficiency  and  teamwork.  Teamwork  metrics 
included  estimations  of  collaboration  power  and  collaboration  correlation,  and  new  higher  order  measures  of 
team  collaboration  co-efficiency,  dynamic  efficiency,  and  team  adaptability  proficiency.  DSTO  AOD 
conducted  an  independent  assessment  complementary  process,  including  a  CAS  SME  continuous  assessment 
of  CAS  performance,  captured  on-line  using  a  digital  tablet-based  protocol,  and  administered  validated  self- 
ratings  of  CAS  Taskwork,  Team  Workload  and  Teamwork  during  post-run  de -briefing. 
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Statistical  analysis  (ANOVA)  of  the  DSTL  assessment  subjective  ratings  showed  significant  effects  (p<0.05) 
of  the  Baseline  (B),  Threshold  (T)  and  Objective  (O)  architectures  as  follows: 

•  Activity  phase  demand:  Mitigation  (B<T);  Dissemination  (B<OT)  p<0.05  (NB  Evaluation  (B<T) 
p<0.09;  Acknowledgement  (B<T)  p<0.08). 

•  4T  Change  demand:  Task,  Threat  (B<OT)  p<0.01. 

•  Workload:  Time  pressure  (B<OT)  p<0.01;  Mental  effort,  Stress  (B<T)  p<0.05. 

•  SA:  Understanding  (T<0)  p<0.01 . 

•  DQ:  Confidence  (T<0),  Effectiveness  (BT<0),  Timeliness  (B<TO)  p<0.001. 

•  Dynamic  Efficiency:  Reward/Effort  (T<0),  Rate  (B<0)  p<0.05;  Sustainability  (BT<0)  p<0.001. 

•  Team  Work:  Co-ordination,  Co-operation,  Collaboration,  Communication,  Leadership,  Power  (BT<0) 

p<0.001. 

•  System:  Confidence  (T<0)  p<0.01;  Reliability  (B<0)  p<0.05  (NB  Usability  (T<0)  p<0.07). 

In  summary,  there  was  some  evidence  that  Baseline  architecture  evoked  lower  ratings  of  activity  phase  and 
change  demand,  and  workload.  Importantly,  there  was  very  strong  evidence  that  the  Objective  architecture 
evoked  significantly  higher  ratings  of  SA,  DQ,  Dynamic  Efficiency  and  Team  Work.  The  consistently  high 
ratings  for  Team  Work  with  the  Objective  architecture  were  the  strongest  differentiator  from  the  Baseline  and 
Threshold  architectures.  This  is  evidence  that  the  enhancement  of  collective  effort  afforded  by  advanced 
DAMM  technology  is  probably  the  strongest  factor  underpinning  the  improvements  in  adaptability 
proficiency  and  performance. 

The  results  of  the  DSTL  assessment  contrasting  the  individual  trial  runs  are  summarised  in  Table  13-3  below. 
Improved  adaptability  proficiency,  followed  by  improved  dynamic  efficiency,  is  hypothesised  as  directly 
underpinning  improvements  in  performance  and  effectiveness  on  dynamic  missions.  Guided  by  this  assertion, 
in  Table  13-3,  the  data  for  the  individual  eight  trial  mission  runs  are  presented  in  the  position  order  (P  1-8)  of  the 
adaptability  proficiency  ratings  provided  by  White  Force  SME  assessors.  The  summation  total  of  the  adaptability 
proficiency  ratings  are  reported  first,  followed  by  with  the  number  of  observations  contributing  to  the  total. 
Then,  target  performance  (+ve/-ve)  is  shown,  followed  by  an  estimation  of  Decision  Flow  (Decision  points,  run 
time,  decision  tempo),  Dynamic  Efficiency  (SME  Observer  and  Participant  ratings  means),  and  Collaboration 
Co-efficiency  (Power  and  Correlation  estimates).  Trial  runs  with  the  Objective  (OB)  architecture  occupied 
Pl/2/3/4/6.  Threshold  (TH)  architecture  runs  are  at  P5  and  P7.  The  Baseline  (BA)  architecture  tested  on  Run  1 
provided  the  lowest  adaptability  proficiency  ratings  and  thus  was  positioned  at  P8.  The  pattern  of  adaptability 
proficiency  results  are  generally  supported  by  the  metrics  of  decision  flow,  dynamic  efficiency  and  collaboration 
co-efficiency.  Run  order,  with  associated  learning  effects,  and  Architecture  condition,  were  partially  confounded. 
Notwithstanding,  the  pattern  of  results  overall  is  consistent  with  superiority  for  the  Objective  architecture. 
The  pattern  of  target  performance  scores  is  complex  and  less  clearly  related  to  the  architecture  condition. 
Four  MC  positions  were  tested:  FWSS;  FW2S;  RWAH;  E3.  MC  position  was  partially  confounded  with 
architectures  and  run  order.  The  effect  of  MC  position  on  adaptability  proficiency  was  complex.  The  MC  role 
requires  good  communications  and  SA.  On  aggregate,  based  on  adaptability  proficiency,  the  FW  positions 
appeared  most  favoured,  and  the  E3  the  least  favoured  positions  (FWSS  Pl/6;  FW2S  P2;  RWAH  P3/7; 
E3  P4/5/8).  The  Objective  architecture  provided  relatively  good  adaptability  proficiency  with  the  MC  in  all  four 
positions,  consistent  with  good  communications  and  SA.  Evidence  from  specific  events  and  interventions  on 
individual  runs  showed  benefits  of  distributed  and  adaptive  decision  making,  afforded  most  by  the  networked 
Objective  architecture. 


RTO-TR-HFM-170 


13-19 


UK-1:  DYNAMIC  AIRBORNE  MISSION  MANAGEMENT 


ORGANIZATION 


Table  13-3:  Summary  of  Results  from  the  2nd  US-UK  NCSC  /  Strike  Warrior  II  SE  Trial. 


p 

Mission 

Adaptability 

Proficiency 

Targets 

Decision  Flow 

Dynamic 

Efficiency 

Collab 

Coeff 

Total 

Obs 

+ 

- 

Pts 

Time 

Tmp 

Obvr 

Part 

Pwr 

Corr 

1 

OB-FWSS-V3-R8 
Bridge  Stop  West 

48 

8 

4 

0 

36 

26:00 

0.43 

6.00 

5.43 

5.71 

5.05 

2 

OB-FW2S-V3-R7 
Ugly  North  South 

46 

7 

3 

2 

23 

21:15 

1.22 

6.00 

5.78 

6.00 

6.27 

3 

OB-RWAH-V3-R5 

Late  West 

39 

6 

5 

0 

23 

32:30 

1.25 

6.00 

5.11 

5.56 

5.82 

4 

OB-E3-V2-R4 

Bridge  Stop  North 

27 

4 

4 

0 

18 

28:00 

1.33 

5.80 

5.37 

5.62 

5.85 

5 

TH-E3-V3-R6 

Wrong  Bridge 

25 

6 

3 

2 

25 

34:00 

1.22 

4.00 

4.33 

3.67 

4.96 

6 

OB-FWSS-V2-R2 
Pick-up  JTAC 

24 

4 

4 

1 

12 

34:35 

2.53 

5.80 

4.87 

5.37 

6.85 

7 

TH-RWAH-V2-R3 
Support  TIC 

25 

4 

4 

0 

13 

41:45 

3.12 

5.60 

5.12 

5.12 

5.37 

8 

BA-E3-V1-R1 

Abort  Hard  Stop 

7 

3 

2 

2 

14 

26:15 

1.52 

4.80 

4.62 

4.12 

4.42 

13.8.2  DAMM  CCD  UAS  SE  Technical  Demonstration,  April  2011  -  UAS  Co-ordination 
with  RW  SH,  FW  TFJ,  C2  and  FAC/JTAC 

Under  DAMM  CCD  Phase  3,  work  was  undertaken  with  Thales  to  develop  a  UAS  reconnaissance  capability  for 
integration  and  demonstration  in  DAMM  trials,  based  on  the  UK  WK  TUAV,  thereby  de-risking  near-term  UAS 
exploitation  of  DAMM  capability.  Thales  provided  a  representation  of  the  WK  GCS  and  a  representation  of  the 
WK  Full  Motion  Video  (FMV)  feed  using  a  Remote  Viewing  Terminal  (RVT)  for  demonstration  at  QinetiQ  in  a 
fully  immersive  environment.  The  UAS  demonstration  scenario  was  based  on  the  Joint  Warrior  NDZ  scenario 
used  for  the  2nd  Joint  US-UK  SE  Trial,  centred  on  the  Invergarry  /  Fort  Augustus  /  Loch  Lochy  area  of  Scotland. 
The  scenario  involved  a  pre-planned  deliberate  operation,  with  a  High  Value  Target  (HVT)  meeting  at  a  known 
site,  RW  SH  inserted  capture,  and  a  co-ordinated  FW  TFJ  strike  on  arms  cache.  The  assets  and  tasking  were  for 
a  WK  UAS  over-watch  on  meeting  site,  FAC/JTAC  control  of  FW  TFJ  strike,  RW  SH  inserts  ground  forces  to 
effect  capture,  FW  TFJ  time  co-ordinated  to  attack  while  RW  SH  on  ground,  and  C2  overall  co-ordination  and 
control.  The  DAMM  mission  enabling  tools  involved  were  OpTEAM,  TacTEAM/TDSS,  OMPS  and  WK  GCS. 
The  following  event  flow  was  successfully  demonstrated  in  the  QinetiQ  SE  facilities: 

•  Initial  asset  disposition  as  pre-plan,  GARS  airspace  pre-allocated. 

•  HVT  arrives,  3  vehicles  scatter  -  WK  reports  to  C2;  HUMINT  reports  SIED  north,  HVT  west, 
Arms  transfer  south. 

•  C2  calls  abort  -  RW  to  hold;  TFJ  to  hold;  FAC/JTAC  relinquishes  airspace. 

•  C2  re-tasks  WK  to  follow  SIED  -  WK  re-plans;  WK  publishes  route. 

•  C2  re -plan  -  C2  assigns  WK  airspace;  C2  re-tasks  RW  to  pursue/stop  SIED;  C2  re-tasks  TFJ  to  delay/ 
obstruct  HVT. 

•  Assets  execute  re -plan  -  Shared  SA  through  GARS,  route,  PPLI  enables  distributed  re-planning  and 
co-ordination  of  individual  taskings. 


13-20 


RTO-TR-HFM-1 70 


dMNATO 
WP  I  OTAN 


UK-1:  DYNAMIC  AIRBORNE  MISSION  MANAGEMENT 


In  a  second  phase  of  work,  the  QinetiQ  HOVERS  SE  was  used  to  demonstrate  the  UAS  RVT  supporting 
representative  RW  SH  operations  in  an  immersive  environment,  with  the  Joint  Warrior  NDZ  scenario  centred 
near  Fort  Augustus,  north  of  Loch  Lochy  in  Scotland.  In  the  UAS  RVT  demonstration,  the  RW  SH  operated 
with  serving  military  aircrew,  was  tasked  with  short  movement  of  ground  troops  to  a  known  Landing  Site 
(LS)  to  create  and  man  a  temporary  road  block,  searching  for  small-scale  movement  of  arms  caches. 
Insurgents  were  known  to  have  4  armed  “technicals”.  In  the  event,  a  moving  vehicle  departed  the  road, 
and  arrived  at  the  LS,  co-incident  with  a  White  Force  SAFIRE  inject  causing  an  RW  SH  direction  change  en 
route  close  to  the  LS.  Two  runs  trial  were  flown,  each  with  a  pre-loaded  route  on  an  OMPS-based  hand  held 
DIGIMAP  system.  The  first  run  was  performed  without  the  RVT  capability.  The  second  run  was  flown  with 
the  RVT  capability.  On  the  RVT  run,  a  TUAV  air  vehicle,  controlled  from  the  WK  GCS,  was  positioned  to 
give  a  virtual  RVT  view  of  the  LS.  The  participant  aircrew  assessments  reported  that  Crew  Resource 
Management  issues  concerning  information  interpretation  were  dependent  on  the  individual  aircraft  receiving 
RVT,  i.e.,  AH  direct  feed;  RW  SH  to  crewman  or  off-board  UAS  image  analyst;  live  feed  to  SH  embarked 
troops.  It  was  considered  that  objective-based  tasking  (e.g.,  point  sensor  at  location  X)  would  most  likely  be 
more  appropriate  than  attempting  direct  control  of  one  or  more  UAS.  Generally,  SA  was  considered  to  be 
enhanced  with  live  positional  information  from  UAS  in  close  proximity. 

13.8.3  RE314  MET&D  Joint  US-UK  SE  Trial,  July  2011  -  Multiple  UAS  Co-ordination  with 

RW  SH,  FW  TFJ,  C2  and  FAC/JTAC 

A  3rd  in  series  Joint  DAMM  SE  Trial  involving  USAF  AFRL  was  conducted  under  the  US-UK  NCSC  /  Strike 
Warrior  II  PA  at  QinetiQ  Famborough  in  July  2011,  with  a  focus  on  multiple  UAS  co-ordination.  USAF 
AFRL  provided  the  MUSCIT  /  Vigilant  Spirit  multiple  UAS  system  and  GCS  for  DAMM  SE  integration  and 
testing.  The  trial  involved  UK  MOD  serving  military  operators  and  USAF  aircrew,  including  US  and  UK 
UAV  operations  specialists.  The  scenario  comprised  the  following  elements: 

•  AWACS  E3  Airborne  C2  with  OpTEAM; 

•  2  FAC/JTAC  with  digital  targeting  technology  (DESCAT  and/or  BATMAN); 

•  1  FW  TFJ  Ground  Attack  aircraft  with  TacTEAM/TDSS; 

•  1  RW  SH  using  an  immersive  simulator  (HOVERS)  with  networked  DIGIMAP;  and 

•  4  Reconnaissance  UAS. 

USAF  AFRL  provided  3D  audio  cueing  capability  for  moving  target  designation  by  FAC/JTAC  and  UAS. 
White  Force  /  Trial  Control  were  provided  with  networked  OpTEAM  as  SA  and  trials  management  tool. 

The  trial  was  based  on  Joint  Warrior  NDZ  scenario  operations  centred  on  the  Invergarry  /  Fort  Augustus  /  Loch 
Lochy  area  of  Scotland,  with  increased  UAS  involvement  in  observation  and  reconnaissance.  The  scenario 
involved  a  pre-planned  deliberate  operation,  with  a  High  Value  Target  (HVT)  meeting  at  a  known  site,  RW  SH 
inserted  capture,  and  a  co-ordinated  FW  TFJ  strike  on  arms  cache.  The  assets  and  tasking  were  for  a  SF  FAC 
observation  and  UAS  over- watch  on  an  adjacent  urban  area  for  known  HVTs,  in  addition  to  the  meeting  site, 
FAC/JTAC  control  of  FW  TFJ  strike,  RW  SH  inserts  of  ground  forces  to  effect  capture,  FW  TFJ  time 
co-ordinated  to  attack  while  RW  SH  on  ground,  and  C2  overall  co-ordination  and  control.  Trial  White  Force 
inserted  information,  events,  threats  and  target  behavioural  changes  requiring  real-time  re-planning  and 
co-ordination.  The  trial  took  place  over  a  period  of  5  days,  following  a  4  weeks  of  extensive  technical 
development,  integration,  and  system  and  SE  testing,  and  progressive  scenario  refinement.  After  completion  of 
role  specific  training,  the  trial  runs  were  conducted  in  two  phases,  with  the  first  phase  comprising  tool  use-case 
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vignettes  and  the  second  phase  end-to-end  scenario  runs.  All  use  cases  and  end-to-end  runs  involved  active 
operator  participation  and  decision  making,  post  run  de-briefing  and  performance  measurement.  DAMM  tools 
were  available  for  use  on  all  the  trial  runs.  The  use-case  runs  provided  additional  training  and  focussed 
performance  data  on  DAMM  tool  usage.  Additionally,  the  use-case  runs  increased  participant  familiarisation 
with  the  DAMM  tools  applicability  and  benefits,  in  readiness  for  the  end-to-end  scenario  runs.  The  scenario  runs 
were  developed  progressively  with  changing  and  evolving  complexity  to  maintain  operator  interest,  engagement 
and  to  challenge  DAMM  competencies  and  tool  usage.  In  total,  6  use-case  vignettes  and  4  end-to-end  scenario 
runs  were  completed.  The  DSTL  trial  assessment  included  participant  self-ratings  and  observer  SME  ratings 
obtained  for  use  cases  and  end-to-end  runs  during  post-run  de-briefing  sessions.  Subjective  ratings  (7-point  Low- 
High  anchors,  Likert  scales)  were  obtained  for  57  trial  parameters  in  total. 

Participants  provided  CMDM  self-ratings  on  17  parameters  for  individual  identified  critical  decisions: 

•  Workload:  WL  Time  Pressure,  WL  Mental  Effort,  WL  Stress,  Team  Workload. 

•  Re -plan  Task  Load:  Re-plan  Decision  (Recognise-Evaluate-Mitigate),  Re-plan  Action  (Disseminate- 
Ackno  wledge-Execute-Report) . 

•  Situation  Awareness:  SA  Demand  on  Attentional  Resources,  SA  Supply  of  Attentional  Resources,  SA 
Understanding. 

•  Decision  Quality:  DQ  Confidence,  DQ  Survivability,  DQ  Effectiveness,  DQ  Timeliness. 

•  Performance:  Task  Performance,  Tools  Utility,  Adaptability  Proficiency,  Probability  of  Mission  Success. 

Participants  provided  ratings  on  DAMM  technical  system  performance  for  25  items: 

•  Assessment  of  the  relative  usability  of  information  and  tools  -  22  items  in  total  divided  into  the  6 
categories  (Comms,  Route,  Position,  SA,  Task,  UAS  Services). 

•  Technical  System  General  Performance:  Confidence,  Reliability,  Usability. 

Observer  SMEs  recorded  observations  and  events  with  associated  mission  times,  and  provided  ratings  on  the 
15  parameters: 

•  Decision  Quality:  DQ  Confidence,  DQ  Survivability,  DQ  Effectiveness,  DQ  Timeliness. 

•  Teamwork:  Communication,  Shared  SA,  Leadership,  Support,  Team  Workload. 

•  Performance:  Task  Performance,  Collaboration,  Influence  Power,  Adaptability  Proficiency,  Probability 
of  Mission  Success. 

Analysis  of  the  results  (ANOVA)  comparing  Observer  and  Participant  ratings  showed  significant  differences 
in  the  ratings  of  DQ  Survivability,  Effectiveness  (p<0.01)  and  Timeliness  (p<0.05),  Adaptability  Proficiency 
(p<0.05),  and  Probability  of  Mission  Success  (p<0.01),  with  significantly  higher  ratings  provided  by 
observers.  Comparisons  between  ratings  for  decision  events  showed  significant  differences.  Observer  data 
showed  significant  differences  between  ratings  of  decision  events  within  end-to-end  runs  for  Shared  SA  and 
Team  Workload  (p<0.05),  Adaptability  Proficiency  (p<0.01)  and  Probability  of  Mission  Success  (p<0.05). 
The  Comparisons  between  the  four  end-to-end  runs  showed  a  pattern  of  higher  ratings  for  the  2nd  trial  run. 
Overall,  these  results  provide  evidence  in  support  of  the  use  of  Observer  SME  ratings. 

Demand  for  UAS  over-watch,  reconnaissance  and  observation  services  varied  continuously  and  reached  high 
levels  during  dynamic  phases  and  events.  Deployment  of  the  multiple  UAS  was  stretched  by  the  scenario. 
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Generally,  the  UAS  were  deployed  effectively  and  efficiently  via  the  MUSCIT  /  Vigilant  Spirit  GCS,  with  the 
UAV  operator  assessment  ratings  indicating  good  operator  engagement  and  SA,  and  contributing  significantly 
to  maintaining  mission  SSA  and  command  flow.  Scenario  emersion  and  workload  was  observed  to  be  high  for 
both  the  UAS  GCS  operator  and  JTAC/FAC.  This  was  confirmed  by  AFRL-provided  real-time  physiological 
measures.  Demand  for  UAS  over- watch,  reconnaissance  and  observation  services  varied  continuously  and 
reached  high  levels  during  dynamic  phases  and  events.  Deployment  of  the  multiple  UAS  was  stretched  by  the 
scenario.  Generally,  the  UAS  were  deployed  effectively  and  efficiently  via  the  MUSCIT  /  Vigilant  Spirit 
GCS,  indicating  high  operator  engagement  and  SA.  UAS  contributed  significantly  to  maintaining  mission 
SSA  and  command  flow.  Scenario  emersion  and  workload  was  high  for  UAS  GCS  operator  and  JTAC/FAC. 
This  was  confirmed  by  AFRL-provided  real-time  physiological  measures. 

13.8.4  DAMM  CCD  RW  Flight  Trial,  August  2011  -  RW  AH  and  RW  SH  Co-ordination 
with  FW  TFJ,  C2,  UAS  and  Deployed  FAC/JTAC 

This  trial  was  an  airborne  demonstration  of  the  Rotary  Wing  elements  of  previous  RE314  SE  trials  with  a 
deployed  FAC/JTAC  using  the  Joint  Warrior  SDZ  operating  out  of  AAC  Middle  Wallop  and  Westdown 
Camp.  Two  trial  missions  were  flown  as  training  sorties,  followed  by  4  trial  sorties,  flown  with  or  without  the 
networked  architecture,  conducted  over  a  period  of  four  days.  This  was  followed  by  a  fully  networked  visitor 
day  demonstration.  All  trial  runs  involved  DSTL  provided  performance  measurement.  The  scenario  involved 
the  following  components. 

•  2  deployed  RW  Lynx  aircraft  with  networked  DIGIMAP; 

•  1  deployed  RW  Apache  AH  (Day  4  only); 

•  1  deployed  FAC/JTAC  with  digital  targeting  technology  (DESCAT); 

•  1  simulated  AW  ACS  E3  Airborne  C2  node  with  OpTEAM; 

•  1  simulated  FW  TFJ  Ground  Attack  aircraft  with  TacTEAM/TDSS;  and 

•  1  simulated  Reconnaissance  WK  TUAV. 

The  DSTL  provided  trial  assessment  included  participant  self-ratings  and  observer  SME  ratings  obtained 
during  post-run  de-briefing  sessions.  Subjective  ratings  (7-point  Low-High  anchors,  Likert  scales)  were 
obtained  for  trial  parameters  using  the  protocols  developed  for  the  July  2011  Joint  US-UK  SE  trial. 

Participants  provided  CMDM  self-ratings  on  17  parameters  for  individual  identified  critical  decisions: 

•  Workload:  WL  Time  Pressure,  WL  Mental  Effort,  WL  Stress,  Team  Workload. 

•  Re -plan  Task  Load:  Re-plan  Decision  (Recognise-Evaluate-Mitigate),  Re-plan  Action  (Disseminate- 
Acknowledge-Execute-Report) . 

•  Situation  Awareness:  SA  Demand  on  Attentional  Resources,  SA  Supply  of  Attentional  Resources, 
SA  Understanding. 

•  Decision  Quality:  DQ  Confidence,  DQ  Survivability,  DQ  Effectiveness,  DQ  Timeliness. 

•  Performance:  Task  Performance,  Tools  Utility,  Adaptability  Proficiency,  Probability  of  Mission  Success. 

In  addition,  participants  provided  ratings  of  the  relative  usability  on  DAMM  information  and  tools  performance 
for  13  items  in  6  categories  (Comms,  Route,  Position,  SA  tools,  Task). 

Observer  SMEs  recorded  observations  and  events  with  associated  mission  times,  and  provided  ratings  on  the 
15  parameters: 
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•  Decision  Quality:  DQ  Confidence,  DQ  Survivability,  DQ  Effectiveness,  DQ  Timeliness. 

•  Teamwork:  Communication,  Shared  SA,  Leadership,  Support,  Team  Workload. 

•  Performance:  Task  Performance,  Collaboration,  Influence  Power,  Adaptability  Proficiency,  Probability 
of  Mission  Success. 

In  addition,  SME  observers  provided  ratings  of  the  relative  usability  on  DAMM  information  and  tools 
performance  for  13  items  in  6  categories  (Comms,  Route,  Position,  SA  tools,  Task).  Analysis  of  the  trial 
results  showed  significant  confounding  of  architecture  conditions  due  to  strong  training/sortie  order  effects. 
Notwithstanding,  for  the  networked  sorties,  there  was  evidence  of  a  consistent  pattern  of  lower  workload 
ratings,  together  with  significant  improvements  (p<0.05)  in  decision  quality,  team  task  performance,  and 
adaptability  proficiency.  All  the  DAMM  tools,  apart  from  1st  Cut  Time  Distance  Fuel  and  Route  Timing 
Information  scored  very  high  on  both  Usability  of  Information  and  Tools  and  Utility  of  Functional  Purpose. 

In  conclusion,  the  results  of  the  trial  provided  high  TRL  evidence,  proof  and  validation  of  the  efficacy  of  the 
DAMM  networked  architecture  and  tools,  of  the  DAMM  principles,  interfaces  and  interactions,  and  of  the 
CONOPS  and  TPPs,  integrated  with  a  representative  synthetic  UAS  reconnaissance  capability  component, 
through  testing  with  real  aircraft  under  realistic,  live  flight,  operational  mission  conditions. 

13.9  LESSONS  LEARNED 

13.9.1  Dynamic  Airborne  Mission  Management 

Evidence  from  the  UK  MOD  DAMM  research  programme  of  technical  demonstrations  and  test  and  evaluation 
trials  on  advanced  digital  networking  and  mission  enabling  technologies  provides  robust  verification  and 
validation  of  DAMM  principles,  interfaces  and  interactions  for  delivering  effects  and  providing  stability  and 
dominance  in  a  dynamic  environment.  DAMM  technologies  and  applications  are  shown  consistently  to  deliver 
an  efficient  and  effective  distributed,  collaborative  and  adaptive  mission  capability.  In  summary,  the  DAMM 
programme  provides  evidence  of  the  efficacy  of  the  following: 

•  Advanced  digital  network  technologies,  operating  within  a  multi-tier  C2-MM  architecture  with 
operational  and  tactical  layers,  provides  the  high  capacity  ad  hoc  networking  with  the  agility,  scale  and 
richness  needed  to  maintain  mission  and  command  flow  through  intra-  and  inter-flight  communication, 
cross-domain  (air  and  land)  integration,  and  integrated  operational  C2. 

•  Mission  enabling  technologies,  coupled  with  advanced  digital  networking,  provide  the  tools  needed 
for  intra-  and  inter-flight  mission  management,  cross-domain  (air  and  land)  dynamic  co-ordinated 
planning,  and  dynamic  battle  management: 

•  Airborne  re -planning/modification  of  a  pre-planned  mission  in  response  to  dynamic  events. 

•  Airborne  plan  formulation  for  missions  which  cannot  be  planned  in  advance. 

•  Airborne  co-ordination  and  de-confliction  of  multiple  packages  with  a  mission  and  between 
multiple  missions. 

13.9.2  UAS  Integration 

MoD  requirements  for  a  UAS  DAMM  enabled  capability  are  the  need  for  flexibility,  agility  and  persistence 
for  over-watch  support  and  targeting.  To  achieve  these  requirements,  the  following  is  needed: 

•  Ability  to  co-ordinate  airspace  and  missions  between  manned  and  unmanned  platforms. 
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•  Ability  to  rapidly  task  UAS. 

•  Ability  to  rapidly  exploit  the  output  from  UAS. 

•  Support  improved  Shared  Situational  Awareness  (SSA)  with  UAS. 

This  involved  the  identification,  specification,  development  and  implementation  of  DAMM  Information 
Exchange  Requirements  (IER)  and  Interface  Control  Documents  (ICD)  for  UAS  requirements  for  airspace 
management,  route  data  and  Plan  Position  Location  Information  (PPLI). 

In  order  to  address  the  requirements  for  airspace  management,  under  the  DAMM  programme,  MoD  have 
developed  an  electronic  version  of  the  Global  Area  Reference  System  (GARS)  for  airspace  management. 
The  reason  for  this  is  that  GARS  is  the  currently  accepted  standard  used  by  NATO  for  managing  theatre 
airspace.  Air  Battle  Managers  currently  allocate  airspace  to  operational  users  using  height-banded  air  cells 
communicated  by  voice,  and  recorded  manually  by  UAS  operators  by  marking-up  charts.  Electronic  GARS 
offers  user  benefits  for  speed,  accuracy  and  electronically  assisted  de-confliction.  Under  DAMM,  electronic 
integration  provides  seamless  communication  across  the  operational  and  tactical  tiers  from  airborne  C2 
(E3  AW  ACS),  through  effector  platforms,  down  to  ground-based  co-ordinators  (FAC/JTAC).  For  rapid  tasking 
of  UAS,  and  for  interoperability  with  manned  platforms,  under  the  DAMM  programme  MoD  have  extended  the 
use  of  standard  data  links  and  message  protocols  to  UAS  taskings.  These  include  Link  16/J  Series  messages  for 
Operational  UAS  (OUAS),  and  Improved  Data  Modem  (IDM)  for  tactical  UAS.  For  rapid  exploitation  of  UAS 
output  in  effector  platforms  (RW,  FW),  under  the  DAMM  programme  MoD  have  proposed  use  of  the  K04.17 
Variable  Message  Format  (VMF)  imagery  message  for  passing  still  images  around  the  DAMM  network. 
This  message  is  used,  both  over  IDM  and  over  Linkl6,  in  support  of  Joint  Strike  Fighter  (JSF).  In  addition  to  the 
above,  in  support  of  shared  SA  with  networked  UAS,  under  the  DAMM  programme,  MoD  have  extended  the 
use  of  route  data  and  Plan  Position  Location  Information  (PPLI).  This  is  achieved  through  an  adaptation  of  the 
K05.1  VMF/IDM,  J2.2  Linkl6  messages  for  passing  geo-location  information.  For  route  plan  messages, 
MoD  has  adopted  the  J1 6.1  J  Series  message  send  over  both  Linkl6  and  IDM. 

A  key  thrust  of  the  UAS  DAMM  integration  work  was  the  investigation  of  the  utility  of  direct  Full  Motion 
Video  (FMV)  feeds  from  platforms  such  as  WK  and  other  ROVER  compatible  sensors,  such  as  Reaper  and 
Litening  targeting  pods,  into  cockpits  such  as  Julius  Chinook  RW  SH.  This  would  allow  the  Julius  Chinook 
crew  to  task  a  sensor  carrying  platform,  direct  from  the  mission  system,  such  as  the  Onboard  Mission 
Planning  System  (OMPS).  Such  tasking  could  provide  co-ordinated  over-watch  and  to  receive  live  in-cockpit 
FMV  during  critical  moments  of  operations,  such  as  heli-borne  insertion  and  extraction  of  troops.  Such  a 
capability  would  significantly  improving  SA  and  therefore  survivability.  Evidence  obtained  from  DAMM  of 
the  utility  of  in-cockpit  FMV  directly  supports  procurement  decisions  for  such  a  capability  in  addition  to 
de-risking  implementation  by  addressing  HMI  and  other  integration  aspects. 

Refer  to  the  Annex  A  in  the  report  for  further  detailed  information  on  DAMM  UAS  Lessons  Learned. 


13.10  CONCLUSIONS 

The  DAMM  SE  and  flight  trials  provide  evidence  with  high  levels  of  proof  for  real  benefits  of  a  full  suite  of 
collaborative  decision  support  tools  across  all  three  tiers  of  the  networked  dynamic  C2-MM  architecture. 
This  work  contrasting  DAMM  architectures  and  networked  collaborative  decision  making  tools  has  shown 
that  as  data  rate,  confidence  and  context  increase/improve  nodal  (micro)  and  system  (macro)  C2  OODA  loop 
activity  transposes  from  slow  serial  to  concurrent-NRT  speeds.  Consequently,  kill-chain  timeline,  fratricide 
and  collateral  incidents  should  reduce,  as  illustrated  in  Figure  13-10  below. 
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ORGANIZATION 


Figure  13-10:  COODA  Fusion. 


13.11  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

Further  trials  are  needed  to  examine  the  effects  of  highly  networked  C2-MM  architectures  and  MC  positions 
on  crew  decision  making  with  increasing  capability  provided  by  UAS.  Work  is  needed  to  identify  emerging 
CONOPS  and  architectures  associated  with  highly  networked  and  distributed  mission  systems  involving 
manned  aircraft  and  UAS.  User  requirements  need  to  be  identified  and  validated  for  tools  to  improve  dynamic 
mission  efficiency  and  adaptability  and  to  enable  improved  tactical  mission  battle  space  integration,  across 
capabilities.  In  particular,  work  is  needed  to  focus  on  proactive  decision  support  tools  in  support  of  improved 
anticipatory  decision  making.  We  need  to  improve  understanding  of  the  effects  on  mission  command  flow  and 
locus  of  critical  mission  decision  making,  such  as  the  balance  of  centralized  versus  distributed  decision 
making.  Changes  in  decision  making  strategies  are  likely  towards  more  decentralized,  distributed  decision 
making  afforded  by  networked  collaborative  decision  support  tools. 
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14.1  DATE 

6  May  2010. 


14.2  LOCATION 

Camp  Atterbury,  Indiana,  USA. 


14.3  SCENARIO/TASKS 

The  objective  of  the  MUSCIT  program  is  to  develop,  integrate  and  demonstrate  technology  for  effective  single¬ 
operator  control  of  multiple  Unmanned  Aerial  Vehicles  (UAVs)  conducting  dynamic  tactical  intelligence, 
surveillance,  and  reconnaissance  and  close  air  support  missions.  The  program  has  set  up  a  series  of  spirals,  made 
up  of  simulation  and  flight  tests  (see  Figure  14-1),  to  examine  human  and  system  performance  for  an  assortment 
of  UAV  Reconnaissance,  Surveillance,  and  Target  Acquisition  (RSTA)  tasks  and  operations. 
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Figure  14-1:  MUSCIT  Program’s  Spiral  Approach. 

The  NATO  technology  demonstration  was  conducted  in  conjunction  with  MUSCIT’s  Spiral  2  flight  test  and 
represented  a  sub-set  of  the  tasks  and  conditions  tested.  The  NATO  technology  demonstration  consisted  of  a 
live  flight  demonstration  of  advanced  single-operator,  multi-UAV  control  station  technology  conducting  point 
surveillance  and  vehicle  re-routing  tasks.  The  control  station  technology,  referred  to  as  the  Vigilant  Spirit 
Control  Station  (VSCS)  (see  Figure  14-2)  [1],  was  demonstrated  in  a  four-vehicle  configuration  made  up  of 
two  flight  test  UAVs  (MLB  Bat  3  s)  and  two  simulated  UAVs.  The  main  task  demonstrated  was  point 
surveillance  where  the  control  station  operator  positioned  the  UAVs  and  manipulated  their  gimbaled  sensors 
to  detect  individuals  either  entering  or  exiting  specified  locations  and  reported  whether  or  not  the  individuals 
were  armed  with  a  mock  weapon  (see  Figure  14-3). 
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Figure  14-2:  Flight  Demonstration  of  Vigilant  Spirit 
Control  Station  Performing  Multi-UAV  Operations. 


Figure  14-3:  Point  Surveillance  Actor  Carrying  Mock  Weapon  During  Flight  Test. 
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Secondary  tasks  included  monitoring  radio  calls  and  reporting  vehicle  heading,  power  remaining,  position 
reports,  any  simulated  air  tracks  transiting  the  area  of  operations,  monitoring  and  reporting  system  cautions 
and  warnings,  and  responding  to  vehicle  relocation  requests.  The  point  surveillance  portion  represented  a 
second  test  of  point  surveillance  for  the  MUSCIT  program  to  determine  if  added  technology  and  design 
enhancements  significantly  improved  target  acquisition  and  secondary  task  performance  from  that  found  in 
the  Spiral  1  test.  The  goal  was  to  strive  for  near-equivalent  performance  (e.g.,  percentage  of  targets  detected) 
and  subjective  ratings  (e.g.,  situation  awareness  ratings)  across  the  different  number  of  UAV  conditions. 


14.4  TECHNOLOGIES  EXPLORED 

The  control  station  consists  of  a  desktop  computer  with  two  side-by-side  24”  widescreen  liquid  crystal 
displays  (see  Figure  14-4  and  Figure  14-5),  a  keyboard  and  a  mouse.  The  user  interface  is  comprised  of 
vehicle  and  system  status  information,  a  Tactical  Situation  Display  (TSD),  vehicle  and  payload  controls,  and  a 
sensor  management  area.  The  vehicle  status  area  provides  current  and  commanded  state  information  to  help 
the  operator  maintain  situation  awareness  of  the  UAV(s).  The  TSD  provides  a  2-dimensional  map  or  image 
with  vehicle  locations  and  other  points  of  interest  depicted  on  it.  The  operator  can  directly  control  certain 
UAV  actions  on  the  TSD.  For  example,  the  operator  can  select  a  UAV  and  change  its  direction  of  flight  by 
manipulating  graphic  controls  associated  with  the  UAV  symbol.  The  vehicle  and  payload  controls  provide 
additional  control  options.  For  example,  the  operator  can  place  the  sensor  in  latitude/longitude  slaved  mode 
and  control  the  zoom  level.  The  sensor  management  area  allows  the  operator  to  control  and  view  the  sensor 
feeds,  manipulate  the  presentation  using  digital  video  recorder  and  mosaic  tools,  and  perform  additional 
payload  control. 


Figure  14-4:  Left  Control  Station  Display  with  System  Status  Display, 
Tactical  Situation  Display,  and  Vehicle  and  Payload  Controls. 
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Figure  14-5:  Right  Control  Station  Display  with  Aircraft  Video  and  Sensor  Management  Area. 


14.5  HUMAN  FACTORS  ISSUES  EXPLORED 

The  objective  of  MUSCIT  Spiral  2  was  to  investigate  the  impact  and  demands  associated  with  multi-UAV 
control  within  the  context  of  a  static  and  dynamic  RSTA  mission  scenario  and  establish  a  performance 
benchmark  for  the  control  of  multiple  UAVs.  Elements  of  the  test  included: 

1)  Evaluating  the  value  of  the  operator  control  station  design  enhancements; 

2)  Investigating  the  performance  effects  and  behavioral  adaptation  of  operators  as  the  number  of  UAVs 
being  controlled  and  video  image  streams  being  monitored  increased;  and 

3)  Identifying  opportunities  via  automation,  visualizations,  control  mechanization,  and  employment 
concepts  that  would  enhance  the  feasibility  of  multi-UAV  control. 

The  previous  spiral’s  tests  generally  showed  that  as  the  number  of  UAVs  increased  there  was  an  overall 
decrease  in  the  percentage  of  targets  detected  and  the  associated  performance  in  identifying  whether  or  not  the 
individual  was  armed.  This  downward  trend  was  also  seen  in  the  secondary  task  performance.  In  addition, 
higher  workload  and  lower  situation  awareness  ratings  were  noted.  Technical  challenges  for  the  MUSCIT 
team  to  address  included  improving  attention  management,  sensor  selection  and  control,  and  information 
access.  With  the  objective  data  and  the  subjective  feedback  from  Spiral  1,  the  MUSCIT  team  modified 
portions  of  the  control  station  and  integrated  additional  control,  display,  and  decision  aiding  technologies. 
Additional  technologies  utilized  for  Spiral  2  to  enhance  situation  awareness  and  improve  operator 
performance  included  the  following: 

•  Speech  recognition  and  synthetic  speech  reporting  to  allow  the  operator  to  request  information  of  the 
system  and  have  the  system  retrieve  and  report  the  requested  information.  The  speech  recognition  and 
speech  synthesis  technology  was  included  in  an  attempt  to  reduce  the  overall  visual  demand/load  and 
permit  the  operator  to  focus  more  on  the  sensor  videos  during  the  RSTA  operations. 

•  Sensor  steering  using  the  mouse/cursor  to  allow  a  point  and  click  method  to  steer  the  sensor,  along 
with  zooming  via  the  mouse  scroll  wheel.  The  rationale  for  integrating  the  mouse  sensor  steering  was 
to  reduce  control  time  and  errors  in  aircraft  video  selection/manipulation  and  alleviate  the  need  to 
switch  between  two  input  devices  (sensor  stick  and  mouse). 


RTO-TR-HFM-170 


14-5 


US-1:  MULTI-UAV  SUPERVISORY  CONTROL 
INTERFACE  TECHNOLOGY  (MUSCIT)  DEMONSTRATION 


•  Synthetic  overlays  in  the  form  of  targeting  flags  were  developed  to  permit  the  operator  to  mark 
locations  of  interest  and  have  graphics  overlaid  on  the  sensor  video  to  depict  the  mark.  The  flags  were 
color  coded  to  correspond  to  the  aircraft/video  source  from  which  the  mark  was  made.  The  intent  was 
that  the  synthetic  flags  would  help  enhance  the  operator’s  situation  awareness  in  recognizing 
locations  and  points  of  interest.  The  technology  could  also  serve  as  a  method  to  share  information 
with  others  about  the  battlespace  and  areas  of  interest. 

•  Glyph  symbology  integrated  as  a  sensor  overlay  in  an  attempt  to  reduce  the  visual  scan  requirements. 
The  graphic  contained  vehicle  heading,  vehicle  altitude,  sensor  heading,  communication  data  link, 
and  fuel  state  information. 

•  Caution  and  warning  annunciations  to  convey  important  state  change  information  were  modified  from 
the  Spiral  1  design.  Visual  alerts  were  annunciated  on  the  sensor  video,  as  overlays,  and  aural  alerts 
using  synthetic  speech  to  help  reduce  the  visual  scan  requirements  for  information  retrieval  and 
assessment,  and  help  in  overall  attention  management. 

14.6  UNMANNED  SYSTEMS  USED 

The  MUSCIT  demonstration  used  the  MLB  Bat  3  UAV  (see  Figure  14-6)  equipped  with  Cloud  Cap 
Technology’s  Piccolo  II  autopilot  and  TASE  stabilized  gimbaled  video  camera  (see  Figure  14-7).  The  Bat  3 
version  used  in  the  MUSCIT  flight  demonstration  was  approximately  5  feet  long  with  an  8.5  foot  wingspan. 
As  outfitted,  the  Bat  3  s  provided  the  ability  to  perform  mission,  flight  and  sensor  management  tasks,  which  in 
turn  helped  the  MUSCIT  team  exercise  the  control  station  controls,  displays  and  decision  aids  and  assess 
operator  performance  and  usability  via  flight  tests  and  demonstrations.  For  example,  the  vehicle  could  be 
vectored  to  fly  a  commanded  heading,  follow  a  waypoint  defined  route,  or  loiter  about  a  given  latitude  and 
longitude.  The  sensor  steering  and  management  tasks  could  entail  assigning  a  stare  point  location  based  on 
coordinates,  and  manually  steering  and  zooming  the  gimbaled  video  camera  with  the  control  station  mouse. 
VSCS  was  capable  of  receiving  state  data  and  the  sensor  video  from  each  Bat  3  while  simultaneously 
displaying  multiple  videos  on  the  control  station  displays. 
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Figure  14-6:  Bat  3  UAV. 


Figure  14-7:  Bat  3  with  Gimbaled  Sensor  Deployed. 
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14.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

As  indicated  in  the  table  below  (see  Table  14-1)  the  MUSCIT  technology  demonstration  provided  information 
pertaining  to  the  supervisory  control  technology  design,  development  and  the  approach  for  how  the  technology 
was  tested  and  evaluated.  The  information  was  conveyed  primarily  at  NATO  HFM-170  meetings  and  the 
flight  demonstration  (see  Figure  14-8).  The  events  provided  an  opportunity  to  share  information  on  the  nature 
of  the  supervisory  control  tasks,  operator  interface  technology  and  integration  concepts  that  could  help 
enhance  supervisory  task  performance,  and  evaluation  methods  and  metrics. 

Table  14-1:  MUSCIT  Technology  Demonstration  -  Level  of  Interaction  with  NATO  HFM-170. 


Planning/Design 

Execution 

Analysis 

Communication 

X 

X 

X 

Coordination 

Collaboration 

Figure  14-8:  Launch  of  Bat  3  at  MUSCIT  Technology  Demonstration. 
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14.8  SUMMARY  OF  RESULTS 

In  manipulating  the  number  of  UAVs  (1,2,  and  4),  the  number  of  operators  (1  and  2),  and  the  target  event  rate 
or  tempo  (“low”:  60  seconds  on  average  and  “high”:  30  seconds  on  average),  Spiral  l’s  human-in-the-loop 
simulation  data  led  to  the  following  conclusions: 

•  UAV,  Crew  Size,  and  Tempo  each  had  significant  impact  on  performance; 

•  Reduction  in  crew  size  adversely  affected  performance  as  number  of  UAVs  being  controlled  was 
increased; 

•  Increase  in  event  rate  had  a  significant  impact  on  performance  as  the  number  of  UAVs  being 
controlled  increased; 

•  Secondary  task  performance  dropped  as  a  function  of  number  of  UAVs;  and 

•  Operator  attitudes  relative  to  feasibility,  reasonableness  and  acceptability  was  significantly  influenced 
by  number  of  UAVs. 

The  Spiral  1  flight  test,  which  only  tested  the  number  of  UAVs  and  the  number  of  operator’s  conditions, 
revealed  similar  trends  in  performance  and  subjective  feedback  to  what  was  seen  in  the  simulation. 
The  averages  across  the  conditions  were  generally  lower  in  flight  test  as  compared  to  the  simulation. 
However,  it  should  be  noted  that  the  flight  test  data  was  limited  as  the  number  of  participants  was  only  four 
(Spiral  1  simulation  had  12  participants)  and  some  of  the  test  cells  were  incomplete  due  to  flight  test 
conditions  and  constraints. 

Both  the  empirical  data  collected  and  experimenter  observations  from  Spiral  1  indicated  that  the  visual  load 
and  attention  demands  were  heavily  concentrated  on  the  task  of  sensor  video  inspection,  looking  for  ground 
personnel  and  manipulating  the  sensors  to  determine  whether  or  not  the  detected  personnel  were  armed. 
This  load  increased  with  the  number  of  UAVs  resulting  in  reduced  primary  and  secondary  task  performance. 
The  empirical  findings,  both  objective  and  subjective,  led  the  team  to  the  development  and  integration  of 
additional  technology  and  modifying  the  operator  interface  design  to  help  reduce  the  visual  demands 
(e.g.,  scanning  and  dwell  time)  required  across  the  control  station  displays.  The  goal  was  to  make  pertinent 
task  information  more  easily  accessible  by  placing  the  data  on  or  adjacent  to  the  sensor  management  area  of 
the  control  station  and  provide  alternate  ways  to  become  aware  of  state  changes  and  retrieve  requested 
information  through  multi-modal  methods  (e.g.,  visual  alerts  and  state  information  overlaid  on  the  videos, 
auditory  alerts,  speech  recognition  and  synthesis  to  retrieve  and  report  data). 

Preliminary  analysis  of  Spiral  2’s  simulation  results,  which  had  ten  participants  and  only  employed  the  “high” 
event  rate,  suggests  the  methods  had  a  positive  impact  overall  as  the  multi-UAV  point  surveillance  target 
detection  and  secondary  task  performance  data,  as  well  the  situation  awareness  and  workload  ratings, 
improved  from  Spiral  l’s  simulation  results.  For  example,  the  1-operator,  4-ship  condition  showed  an  11% 
increase  in  target  detection  and  the  secondary  task  performance  increased  by  40%.  Additionally,  across  all  the 
UAV  levels  (i.e.,  1-,  2-,  and  4-UAVs),  Spiral  2’s  China  Lake  Situation  Awareness  ratings  were  better  and  the 
NASA-TLX  Workload  weighted  scores  were  lower  than  seen  in  Spiral  1. 

It  should  be  noted  that  the  trial  times  between  Spiral  1  and  Spiral  2  point  surveillance  simulations  were 
different  with  Spiral  l’s  lasting  approximately  10  minutes  whereas  Spiral  2  was  approximately  8  minutes  in 
length.  The  interval  between  target  events  for  the  primary  task  was  the  same  (30  seconds  on  average)  so  there 
was  a  difference  in  total  number  of  target  events  given  the  total  trial  times  were  different,  while  the  number  of 
secondary  tasks  was  the  same  for  both  spirals.  Another  performance  measure  taken  in  the  point  surveillance 
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task  was  based  on  correctly  reporting  whether  or  not  the  personnel  detected  were  armed.  The  Spiral  2 
simulation  had  a  different  ground  texture  and  weapon  model  than  used  in  Spiral  1  which  could  have  had  some 
effect  on  the  ability  to  identify  whether  or  not  the  personnel  were  armed.  As  a  result,  the  associated 
performance  data  (i.e.,  percentage  of  targets  correctly  identified)  is  not  included  as  part  of  the  comparison. 

The  MUSCIT  Team  experienced  significant  challenges  in  preparing  for  and  conducting  the  Spiral  2  flight  test. 
Weather  and  equipment  problems  hampered  the  ability  to  conduct  the  tests  as  planned.  Only  four  of  the 
planned  six  participants  actually  completed  the  tests.  Furthermore,  due  to  equipment  issues,  only  two  aircraft 
were  available  for  the  data  collection  which  led  the  team  to  employ  two  Bat  3  aircraft  and  two  simulated 
aircraft  for  the  4-UAV  trials.  Additionally,  windy  conditions  for  portions  of  the  data  collection  sessions 
resulted  in  less  than  ideal  test  conditions.  Consequently,  the  data  is  somewhat  suspect  in  fulfilling  the  intended 
analyses  and  drawing  any  firm  conclusions,  but  the  data  showed  similar  trends  to  the  simulation  but  with  a 
sharper  drop-off  in  detection  performance  for  the  4-vehicle  condition,  along  with  generally  less  positive 
subjective  ratings. 


14.9  LESSONS  LEARNED 

As  the  MUSCIT  program  proceeded  from  Spiral  1  to  Spiral  2  the  need  to  re-address  attention  management, 
information  access  and  control  methods  became  apparent.  The  multi-vehicle  control  station  prototype  design 
evolved  from  earlier  applied  research  that  concentrated  primarily  on  scenarios  with  tasks  that  were  typically 
more  sequential  in  nature.  A  priori  knowledge  of  the  targets  (and  their  locations)  and  associated  operator  tasks 
permitted  the  mission  routes  and  operator  action  points  to  be  set  up  in  a  scheduled  and  orderly  fashion. 
The  information  presentation  and  control  methods  supported  the  tasks  and  the  allotted  time  could  be  managed 
to  some  degree  by  the  operator  through  careful  mission  and  route  planning.  The  fact  that  assorted  mission, 
flight,  and  sensor  management  information  was  spread  across  a  large  display  area  did  not  surface  as  a 
significant  issue  in  previous  efforts  and  demonstrations.  The  scenarios  and  RSTA  tasks  examined  in 
MUSCIT ’s  tests  exercised  the  user  interface  in  some  different  ways,  which  in  turn,  highlighted  some  areas  for 
improvement. 

Initially  MUSCIT’ s  point  surveillance  scenario  was  thought  to  be  fairly  low  in  terms  of  complexity  in  that  the 
vehicle(s)  loitered  about  the  known  locations  and  the  operator  would  concentrate  mainly  on  manipulating  the 
sensor(s)  and  reporting  on  the  ground  activity.  However,  given  the  asynchronous  nature  of  the  target  events, 
there  was  a  degree  of  uncertainty  on  what  to  expect  and  when  to  expect  it.  This,  along  with  the  rate  of  target 
events,  compelled  the  operator  to  visually  attend  to  the  sensor  management  area  in  an  attempt  to  detect  and 
identify  the  targets.  Cross-checking  for  system  status  and  tactical  information  in  the  mission  and  flight 
management  areas  could  come  at  the  cost  of  not  detecting  one  or  more  targets.  Therefore,  the  uncertainty  and 
pace  of  these  mission  tasks  highlighted  the  need  for  design  enhancements  to  the  control  station  prototype. 
As  previously  noted,  the  subsequent  spiral  concentrated  on  enhancing  the  design  by  introducing  additional 
technology  and  modifying  the  control  station,  and  improvements  in  performance  were  realized.  The  modified 
control  station  provided  means  to  relieve  some  of  the  visual  demands  and  ease  control,  however,  challenges 
remain  to  enhance  operator  performance  beyond  levels  achieved. 

The  spiral  approach  taken  to  conduct  a  human-in-the-loop  simulation  followed  by  a  closely  aligned  flight 
test  has  afforded  important  insights  into  the  supervisory  control  technology  and  human  performance. 
The  simulations  were  tightly  controlled  experiments  used  to  manipulate  the  number  of  operators  and  UAVs 
and  permitted  a  rigorous  method  to  quantify  human  performance  and  capture  subjective  data.  The  flight  tests 
were  more  constrained  given  the  flight  test  environment  and  resources,  and  it  was  subject  to  more  nuisance 
variables  and  variability  across  the  conditions  and  between  the  participants,  but  it  provided  the  team  a  greater 
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understanding  and  appreciation  for  how  actual  UAV  systems  perform  and  can  be  affected  by  assorted 
environmental  conditions.  The  flight  tests  exposed  the  test  conditions  to  elements  that  may  have  been 
overlooked,  ignored,  or  over  simplified  in  the  simulations.  For  example,  the  wind  and  visual  conditions  were 
not  varied  in  the  simulation  but  in  flight  test  the  conditions  could  vary  across  the  course  of  the  data  collection 
for  a  given  participant,  as  well  as  between  the  participants,  and  could  invoke  different  levels  of  sensor 
management  workload.  Thus,  the  flight  test  brought  a  better  understanding  of  the  real-world  complexities 
associated  with  multi-UAV  supervisory  control  and  can  be  used,  with  the  simulation  findings,  to  derive 
technology  requirements  and  lead  to  further  advancements  in  the  controls,  displays  and  decision  aids,  and  the 
overall  supervisory  control  capability. 

Finally,  given  what  has  been  learned  in  the  MUSCIT  program  to  this  point,  perhaps  “the  problem”  should  be 
examined  from  another  perspective  as  well.  Rather  than  focus  entirely  on  the  operator-to-vehicle  ratio  and 
striving  for  equitable  performance  as  the  vehicles  are  increased,  the  program  might  consider  including  a 
detailed  assessment  of  the  number  and  type  of  tasks  and  sub-tasks  that  must  be  completed  within  an  allotted 
amount  of  time,  with  a  goal  to  effectively  manage  more  tasks,  of  greater  complexity,  and  within  the  mission 
time  “budget”.  In  essence,  try  to  characterize  capability  (e.g.,  mission  effectiveness  and  crew  performance)  as 
a  function  of  task  complexity,  number  of  simultaneous  tasks,  task  sequence,  time  available,  number  of  assets, 
number  of  operators,  etc.  This  more  bottom-up  approach  to  developing  MUSCIT’s  control  station  technology 
could  potentially  complement  the  top  down  approach  by  addressing  the  fundamental  work  components  which, 
in  turn,  might  help  guide  technology  development  and  ultimately  provide  answers  to  how  many  UAVs  and 
operators  are  needed  to  effectively  and  consistently  accomplish  specified  mission  tasks.  In  addition,  rather 
than  use  equitable  operator  performance  and  feedback  across  all  the  data  collected  as  a  goal  when  increasing 
the  number  of  UAVs,  perhaps  the  team  needs  to  more  closely  examine  task  and  sub-task  priorities,  dictated  by 
the  mission  and  the  operator,  and  come  up  with  a  way  to  recognize  and  weight  these  priorities  with  regard  to 
assessing  the  adequacy  and  effectiveness  of  the  entire  system.  Finally,  it  might  be  useful  to  explore  how 
multiple  vehicles  could  be  concentrated  on  a  single  objective  or  a  set  of  serial  objectives,  looking  at  how 
1  -Yi  vehicles  can  be  used  to  more  effectively  accomplish  RSTA  tasks  and  missions.  In  other  words,  determine 
when  it  is  more  effective  to  use  a  team  of  UAVs  on  a  given  objective  versus  one  UAV.  Certain  multi-UAV 
operations  may  be  more  acceptable  in  terms  of  situation  awareness,  workload,  and  performance  levels  when 
the  UAVs  are  concentrated  and  coupled  on  one  common  task  or  objective,  rather  than  disparate  ones. 


14.10  STUDY  CONSTRAINTS/LIMITATIONS 
14.10.1  Flight  Testing  “Live”  and  “Virtual”  UAVs 

For  the  flight  tests,  the  goal  was  to  conduct  the  one,  two  and  four  UAV  case  using  the  Bat  3  UAVs  in  all  the 
conditions.  However,  all  the  Bat  3s  planned  for  were  not  available  due  to  assorted  reasons  for  both  Spiral  1 
and  2  flight  tests  resulting  in  tests  that  involved  a  mixture  of  “live”  and  “virtual”  UAVs,  thus  the  sensor 
imagery  depicted  was  actual  video  and  simulated  video  in  the  four  UAV  cases.  Observations  suggested  that 
the  participants  attended  to  the  systems  differently  at  times.  The  participants  appeared  to  spend  less  time 
watching  and  manipulating  the  simulated  systems  than  they  did  the  “live”  systems.  Part  of  this  may  have  been 
due  to  the  instability  (e.g.,  jitter  and  bounce),  variable  image  quality,  more  frequent  drift  of  the  live  sensor 
about  the  target  location,  and  somewhat  slower  zoom  rates  seen  with  the  actual  sensor.  These  differences  may 
have  attributed  to  the  operator  having  to  manually  control  the  Bat  3  sensor  position  more  frequently  than  with 
the  virtual  system.  The  bottom-line  is  that  there  appeared  to  be  some  difference  in  operator  attention  to  and 
interaction  with  the  live  and  virtual  systems,  which  raises  concerns  with  the  flight  test  data  collected  in  the 
4-UAV  condition. 
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14.10.2  Success  Criteria 

Opinions  have  varied  widely  in  establishing  credible  and  challenging  exit  or  “success”  criteria  for  the  MUSCIT 
program.  How  many  UAVs  should  the  single  operator  control  station  technology  effectively  support?  Under 
what  conditions?  What  are  the  appropriate  measures  of  performance  and  mission  effectiveness,  and  to  what 
levels  should  the  program  strive  for?  One  approach  is  to  try  to  achieve  equivalency  of  performance, 
effectiveness,  and  subject-matter  expert  acceptance.  However,  as  the  effort  and  associated  control  station 
technology  take  on  more  UAVs  and  under  more  complex  conditions,  the  goal  of  reaching  this  equivalency  may 
be  setting  expectations  too  high.  Another  opinion  expressed  is  that  drop-off  of  operator  performance  associated 
with  any  one  UAV  in  the  multi-UAV  operations  should  be  expected  to  some  degree  and  that  the  capability  or 
mission  “success”  is  in  the  aggregate.  More  is  potentially  accomplished  as  more  vehicles  are  introduced. 
However,  this  view  of  the  system  emphasizes  the  benefits  without  addressing  the  potential  costs  (e.g.,  missed 
targets,  mishaps).  How  much  drop-off  in  performance  and  effectiveness  for  any  single  UAV  is  acceptable? 
Given  the  somewhat  exploratory  nature  of  this  domain,  the  MUSCIT  team  chose  to  press  for  equivalency  as  it 
sets  a  measureable  level  to  reach  for,  realizing  it  may  be  very  challenging  to  obtain  with  the  current  state  of  the 
enabling  technologies  and  resources  available. 

14.10.3  Testing  the  System 

In  setting  up  the  point  surveillance  task  and  associated  environment  the  team  was  uncertain  on  what  rate  of  target 
activity  was  appropriate  to  represent  a  meaningful  and  relevant  mission  situation.  Given  there  was  not  a 
prescribed  level  to  simulate  a  “typical”  mission,  the  team  opted  to  implement  a  task  rate  that  would  reduce  the 
likelihood  of  encountering  a  ceiling  effect.  Thus,  the  technology  would  likely  support  the  one-UAV  condition 
fairly  well  with  regard  to  performance,  but  perhaps  not  perfectly  in  all  cases  and,  similarly,  the  2-  and  4-UAV 
cases  would  result  in  less  than  perfect  performance.  In  essence,  the  MUSCIT  team  chose  to  create  a  test  that  was 
likely  too  difficult  to  “ace”.  Some  of  the  participant  feedback  suggested  that  the  event  rate  was  higher  than  they 
typically  encounter  so  perhaps  the  bar  was  set  too  high,  and  the  performance  disparity  seen  between  the  1-,  2-, 
and  4-UAV  conditions  amplifies  too  negatively  on  the  technology  and  overall  capability.  A  “lesser”  test  might 
have  resulted  in  more  equivalent  performance,  effectiveness  and  opinion  across  the  number  of  UAVs. 

14.11  CONCLUSIONS 

The  MUSCIT  program  is  attempting  to  advance  UAV  control  station  technology  to  enable  effective  multi- 
UAV  RSTA  operations  by  a  single  operator.  It  has  developed  a  spiral  approach,  combining  simulation  and 
flight  test,  to  characterize  performance,  empirically  derive  and  refine  technical  requirements,  and  advance  the 
technology  and  overall  capability.  Through  the  approach,  the  MUSCIT  program  has  advanced  control  station 
technology  and  improved  single-operator,  multi-UAV  performance. 

The  MUSCIT  technology  demonstration  for  NATO  HFM-170  consisted  of  single-operator,  multi-UAV  point 
surveillance  using  the  Air  Force  Research  Laboratory’s  Vigilant  Spirit  Control  Station  technology. 
The  demonstration  provided  the  NATO  HFM-170  members  an  opportunity  to  see  the  flight  test  set-up,  control 
station  technology,  and  flight  test  operations.  The  demonstration,  along  with  periodic  meetings,  provided  a 
forum  to  exchange  technical  information  and  discuss  possible  future  collaborations  in  supervisory  control 
research  and  development. 

14.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

In  the  next  spiral,  the  MUSCIT  team  will  increase  mission  complexity  and  afford  operator(s)  more  latitude 
with  respect  to  “managing”  the  mission.  Participants  will  be  afforded  an  extended  training  opportunity  so  they 
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can  become  more  familiar  with  the  array  of  technologies  and  the  various  ways  to  employ  the  controls  and 
displays  during  the  prosecution  of  a  mission.  The  next  spiral  will  be  less  structured  than  the  previous  in  that 
the  participants  will  be  able  to  employ  different  strategies  in  accomplishing  the  mission  tasks,  which  is 
expected  to  provide  additional  insight  into  the  control  station  utility  and  possible  enhancements. 
The  evaluation  will  also  include  an  enhanced  2-operator  control  station  design  to  allow  for  transfer  of  vehicle 
and  sensor  control  between  operators  which  will  permit  the  MUSCIT  team  to  examine  crew  task  allocation 
and  management  methods. 
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15.1  DATES 

The  Delegation  Control  of  Multiple  Unmanned  Systems  (DELCON)  demonstration  was  conducted  on 
24  April  2009. 

15.2  LOCATION 

Recent  research  at  the  U.S.  Army  Aeroflightdynamics  Directorate  (AFDD)  has  focused  on  delegation  control, 
and  delegation  interface  theory,  as  a  solution  for  addressing  the  challenges  of  multi-UAS  control  by  a  single 
operator.  AFDD’s  delegation  control  work  builds  upon  previous  application  of  Playbook®,  developed  by  Miller, 
Goldman,  Funk,  Wu,  and  Pate  [1].  This  research  is  being  addressed  through  two  complementary  tracks: 

1)  A  series  of  empirical  studies  to  test  the  important  tenets  of  delegation  control;  and 

2)  Demonstrations  of  AFDD’s  delegation  control  interface  in  live  flight. 

This  section  describes  the  first  of  these  live  flight  demonstrations. 

The  flight  demonstration  was  conducted  at  the  Military  Operations  on  Urban  Terrain  (MOUT)  site  at  Ft.  Ord, 
California  (Figure  15-1).  The  site  is  located  on  twenty  acres  of  10%  graded  terrain.  The  small  city  mock-up 
contains  33  cinder  block  training  buildings  of  varying  heights  between  1  and  4  stories  high.  Narrow 
alleyways,  winding  gravel  roads,  and  vegetated  surrounding  slopes  characterize  the  city. 
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Figure  15-1:  MOUT  Site  at  Ft.  Ord,  California. 


15.3  SCENARIO/TASKS 

Support  of  troops  in  contact  has  been  identified  as  a  high  priority  and  heavy  workload  mission  for  UAV 
operators  due  to  the  inherent  time  pressure  and  clear  danger  for  troops  on  the  ground.  A  Troops-in-Contact 
(TIC)  mission  for  UAVs  involves  close  air  support  and  surveillance  of  an  area  where  friendly  troops  are 
taking  enemy  fire.  Currently,  a  TIC  may  have  multiple  UAVs  offering  close  air  support  and  surveillance  over 
a  common  target.  In  these  instances,  multiple  operator  teams  control  a  single  UAV  per  team  and  coordinate 
flight  paths  and  airspace  through  Combined  Air  Operations  Center  (CAOC),  using  mIRC  and/or  radio,  as  well 
as  other  Command  and  Control  (C&C)  channels.  Delegation  Control  offers  a  solution  for  multiple  vehicle 
management  and  coordination  over  a  common  target  with  reduced  workload  for  a  single  operator. 

An  appropriate  mission  context  to  justify  the  use  of  heterogeneous  unmanned  systems  was  warranted  to  best 
exercise  Delegation  Control.  The  use  of  multiple  unmanned  assets  for  persistence  on  target  in  dense  urban 
terrain  is  an  understood  necessity.  As  such,  a  TIC  mission,  evolving  into  a  weapons  engagement  with  troop 
re-supply  was  selected  to  showcase  the  capabilities  of  delegation  control. 

Initially,  all  air  and  ground  assets  were  conducting  separate  and  independent  missions.  The  operator  received 
an  incoming  message  communication  that  a  TIC  was  in  progress.  A  TIC  play  was  called  by  the  operator 
through  voice  recognition  control,  resulting  in  all  assets  directed  toward  the  TIC  location.  A  follow-on 
communication  informed  the  operator  of  a  second  enemy  location  and  third  location  requiring  ammunition 
re-supply.  The  operator  responded  by  calling  a  Prosecute  Target  play  and  Quick  Supply  play.  Assets  were 
appropriately  removed  from  the  TIC  play  and  new  asset  allocation  was  accepted  by  the  operator.  All  changes 
were  appropriately  updated  in  the  Play  Status  window.  The  virtual  Shadow  and  Warrior  Alpha  completed  a 
collaborative  weapons  engagement  as  prescribed  by  the  Prosecute  Target  Play.  In  further  detail,  Shadow  and 
Warrior  Alpha  were  directed  to  deconflicted  loiter  patterns  with  payloads  pre-pointed  over  a  common  target 
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and  placed  in  notional  prelaunch  constraints.  The  operator  was  prompted  to  lock-on  the  target  and  entered 
laser  Pulse  Repetition  Frequency  (PRF)  codes.  With  laser  designator  engaged,  the  operator  entered  weapons 
release  codes  and  manually  fired  the  missile.  Upon  missile  impact,  a  pop-up  text  window  prompted  the 
operator  to  select  a  follow-on  action.  The  operator  selected  Conduct  Battle  Damage  Assessment  (with  single 
vehicle).  Consequently,  battle  damage  assessment  was  conducted  by  the  RMAX  (upon  completion  of  Quick 
Supply  play). 

In  parallel,  the  MAX  Rover  and  RMAX  were  en  route  to  the  resupply  location  specified  by  the  operator. 
The  RMAX  flew  in  obstacle  field  navigation  mode  (scripted  behavior)  to  the  quick  supply  load  point. 
The  load  weighed  approximately  15  lbs.  and  was  transported  in  a  sling  extending  5  meters  from  the  bottom  of 
the  aircraft.  Upon  attachment  of  the  sling,  the  RMAX  lifted  the  load  off  the  ground  and  took  off  toward  the 
Drop  Zone  (DZ).  At  this  time,  the  operator  received  intelligence  that  the  DZ  had  been  compromised  and 
should  drop  at  an  alternate  location.  In  response  to  this  communication,  the  operator  modified  the  play 
accordingly.  The  Quick  Supply  play  was  updated  in  real-time  in  the  play  status  window  and  associated  assets 
were  redirected  to  the  new  DZ.  After  scanning  the  DZ  for  obstacles,  the  RMAX  dropped  the  re-supply 
package.  In  coordination,  the  UGV  provided  overwatch  security  and  video  of  the  DZ  and  surrounding  area. 
Once  all  plays  were  completed,  the  assets  returned  to  their  separate  and  independent  missions. 


15.4  TECHNOLOGIES  EXPLORED 

15.4.1  Ground  Control  Station  (GCS)  Hardware 

GCS  hardware  consisted  of  all  computers  required  to  perform  the  demonstration  safely  and  reliably. 
For  safety  measures,  each  unmanned  system  had  a  computer  and  an  assigned  backup  operator  to  monitor 
system  health  and  performance.  In  addition,  a  secondary  backup  operator  monitored  payload  and  telemetry 
functions  for  the  RMAX. 

Primary  MUSIM  software  and  Delegation  Control  interface  resided  on  a  2.4  GHz  Intel(R)  Core™2  Quad 
computer  with  2  GB  RAM.  The  graphics  card  was  an  NVidia  9800GTX+.  A  Samsung  2333SW  monitor  with 
1280  x  720  display  resolution  was  utilized.  Audio  feedback  was  monitored  by  the  operator  using  a  Sennheiser 
PC  136  USB  headset. 

15.4.2  Payload  Hand-Controller 

The  gimbaled  sensor  was  operator-controlled  via  a  3D  Connexion  SpaceExplorer  input  device  (Figure  15-2). 
This  input  device  was  pressure  sensitive  and  required  right/left  and  up/down  twisting  motions  to  control  the 
starepoint  of  the  selected  active  payload.  A  standard  optical  mouse  was  used  for  navigation  of  operator  control 
panels  and  cursor  control  throughout  the  operator  interface  (including  messaging  page). 
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Figure  15-2:  Connexion  SpaceExplorer  with  6  DOF  Control  Movement. 


15.4.3  Multi-UAS  Simulation  (MUSIM)  Software 

MUSIM  software  resided  on  a  Suse  Linux  10.3  operating  system.  Internally  developed  by  AFDD-contracted 
software  engineers,  the  software  suite  was  constructed  utilizing  OpenSceneGraph  for  graphics  and  FLTK  for 
the  graphical  user  interface.  A  simulated  terrain  database  of  Ft.  Ord,  correlated  to  the  real-world  was  created 
using  Creator  Terrain  Studio  2.2  and  Creator  2.5.1.  Terrain  imagery  was  obtained  from  U.S.  Geological 
Survey  aerial  photography.  The  simulation  utilized  10-meter  terrain  data  along  the  MOUT  site  perimeter  and 
1.25-meter  terrain  data  within  the  “city”  at  the  MOUT  site.  Display  of  sensor  imagery  from  the  virtual  UASs 
was  updated  at  60  Hz. 

15.4.4  Voice  Recognition  Control 

A  custom  application  developed  by  the  U.S.  Air  Force  Research  Lab  (Wright-Patterson  Air  Force  Base)  from 
an  integration  of  two  Commercial-Off-The- Shelf  (COTS)  products  was  added  to  MUSIM  software  for 
combined  speech  recognition  and  text-to-speech  capabilities.  The  custom  application  was  comprised  of  SRI 
International’s  DynaSpeak  1.5.32  Speech  Recognizer  and  integrated  with  Cepstral’s  Text-to-Speech  to 
accomplish  voice  recognition  control.  SRI’s  speech  recognizer  was  Linux-based,  offering  speaker  independent 
capabilities  with  natural  language  in  North  American  dialect.  Text-to-speech  was  annunciated  by  the  female 
voice  “Callie”  developed  by  Cepstral  for  Linux  (v5.x). 

Customized  grammar  including  play  commands,  stored  geographic  locations,  and  strike  window  times  were 
defined  for  operator  use  during  the  flight  demonstration.  Integration  between  MUSIM  and  the  voice 
recognition  application  was  accomplished  through  UDP  messages  sent  from  the  speech  recognition  system  to 
MUSIM  as  a  set  of  key  value  pairs  in  a  character  string.  At  a  minimum,  messages  always  contained  a 
“command”  associated  with  a  value  [i.e.,  TIC  (command)  at  Tango  (location  =  value)].  In  cases  where  the 
system  was  unable  to  recognize  a  voice  command,  the  system  issued  a  “No  rec”  statement  via  audio  feedback. 
Operator  voice  commands  were  acknowledged  with  echoed  verbal  confirmations  from  “Callie.” 
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15.4.5  System  Architecture  and  Integration 

Play  origination  began  with  operator  input  to  the  Delegation  Control  interface.  For  RMAX  and  MAX  Rover, 
MUSIM  software  propagated  aircraft/ground  waypoint  and  payload  specifications  to  the  associated  vehicle 
control  system  for  movement  in  the  x,  y,  and  z  axes.  Thus,  complex  collaborative  vehicle  behaviors  were 
decomposed  and  relayed  in  a  series  of  simple  commands  to  the  aircraft  and  ground  vehicle.  In  cases  where  the 
RMAX  control  system  possessed  a  resident  script  for  a  unique  behavior  (e.g.,  obstacle  field  navigation, 
safe  determination  and  landing),  MUSIM  evoked  script  execution.  Command  of  navigation  for  both  RMAX 
and  MAX  Rover  were  executed  through  TCP/IP  messages  generated  by  MUSIM  and  sent  to  the  aircraft  and 
UGV  control  system,  respectively.  For  RMAX  navigation,  Cartesian  coordinates  represented  in  the  MUSIM 
database  were  first  translated  to  UTM  coordinates  and  then  sent  to  the  aircraft  control  system.  For  MAX 
Rover  ground  track  navigation,  MUSIM  software  communicated  Cartesian  coordinates  which  were  translated 
by  the  resident  control  system  to  GPS  lat./long.  coordinates.  Serial  sensor  images  were  sent  in  jpeg  format 
from  RMAX  and  MAX  rover  via  UDP  packets  to  the  MUSIM  computer. 

15.4.6  Delegation  Control  Interface 

The  Delegation  Control  interface  developed  by  U.S.  Army  AFDD  for  this  demonstration  was  comprised  of 
three  separate  display  elements: 

1)  A  digital  moving  map; 

2)  A  Plays  Multi-Function  Display  (MFD);  and 

3)  A  multi-sensor  display  window  (Figure  15-3). 


Figure  15-3:  Delegation  Control  Operator  Interface. 
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A  digital  moving  map  resided  in  a  fixed-size  window  in  the  upper  left  corner  of  the  operator  display.  Mission 
assets  were  depicted  with  vehicle  shape-specific  icons  and  uniquely  color-coded.  Flight  paths  and/or  ground 
tracks  matched  the  color  of  the  related  vehicle  icon.  In  cases  where  dynamic  re -planning  of  a  flight  path 
occurred,  the  intended  flight  path  was  depicted  in  white.  Once  confirmed  by  the  operator,  the  flight  path  was 
displayed  in  the  associated  vehicle’s  color.  Sensor  direction  and  field  of  view  for  each  vehicle  was  displayed 
using  similarly  color-coded  sensor  “whiskers.”  The  sensor  actively  controlled  by  the  operator  displayed  a 
sensor  footprint  highlighted  in  yellow.  Direction  of  aircraft  orbit  was  presented  utilizing  a  clockwise  or 
counter  clockwise  curved  vector  symbol.  Map  navigation  and  zoom  buttons  were  located  along  the  right  side 
of  the  map  display.  Real-time  vehicle  position  was  displayed  and  updated  at  60  Hz. 

Below  the  map  display,  a  Plays  page  was  located  for  the  purpose  of  building,  editing,  and  displaying  the  status 
of  a  play  (Figure  15-4).  Before  describing  a  play  build,  it  should  be  noted  that  development  of  a  multi-play 
library  began  with  the  definition  of  a  single  play.  Plays  had  a  data  structure,  assigned  assets,  and  specific  event 
timing.  Initial  defaults  for  altitude,  loiter  diameters,  vehicle  assignments,  and  sensor  behaviors  were  defined 
prior  to  populating  the  play  library.  These  assignments  were  given  with  regard  to  current  Tactics,  Techniques, 
and  Procedures  (TTPs).  Consideration  of  cost  was  also  important  when  defining  the  plays.  Thus,  vehicles 
navigated  the  shortest  path  to  a  designated  waypoint,  unless  guided  to  an  alternate  route  through  operator 
editing.  In  total,  the  play  library  consisted  of  five  previously  defined  plays. 


Figure  15-4:  Plays  MFD  Page  (e.g.,  Editing  Waypoint  Information  Shown). 


15-6 


RTO-TR-HFM-1 70 


dMNATO 
WP  I  OTAN 


US-2:  TECHNOLOGY  DEMONSTRATOR  10: 
DELEGATION  CONTROL  OF  MULTIPLE  UNMANNED  SYSTEMS  (DELCON) 


The  Plays  page  was  divided  into  three  sections  vertically: 

1)  Play  Builder; 

2)  Play  Modification;  and 

3)  Play  Status. 

The  top  section,  Play  Builder,  contained  the  menus  for  selecting  and  building  a  play  from  the  Play  Library. 
The  middle  section,  or  Play  Modification  section,  displayed  detailed  information  about  a  selected  play  and 
allowed  for  operator  editing  of  navigation  and  sensor  behavior  parameters.  The  bottom  portion  of  the  page 
was  reserved  for  display  of  play  status  and  play  priority  order. 

Delegation  control  of  unmanned  assets  was  accomplished  through  operator  selection  of  a  single  play  from  the 
Play  Library.  To  begin,  the  operator  selected  a  play  from  the  scrolling  menu  of  stored  plays.  Next,  the  operator 
designated  a  location  for  play  execution.  Location  could  be  selected  from  a  list  of  stored  coordinates,  typed 
manually  (e.g.,  numerical  grid  coordinate  entry),  or  designated  by  clicking  on  a  coordinate  point  on  the  digital 
map.  Subsequently,  the  operator  specified  a  time  window  for  play  execution,  to  include  an  option  for  immediate 
action  (i.e.,  now).  In  the  case  of  target  prosecution,  the  time  entry  served  as  a  strike  window  for  earliest  possible 
and  latest  acceptable  time  for  target  strike.  Once  the  operator  specified  a  location  and  time,  a  flight  plan  was 
generated  on  the  digital  map  and  the  play  was  displayed  in  the  Play  Status  window.  In  cases  where  assets  were 
already  in  use  on  an  existing  play,  the  flight  plan  information  box  displayed  the  impact  of  allocating  assets 
between  plays  (i.e.,  RMAX  and  Rover  will  be  removed  from  TIC  for  quick  supply.)  During  simultaneous  play 
execution,  the  flight  plan  information  box  served  to  enhance  automation  transparency  for  the  operator. 
When  simultaneous  plays  competed  for  unmanned  systems  resources,  the  operator  was  required  to  select 
Execute  in  acceptance  of  the  vehicle  allocation  before  the  new  play  initiated.  No  alternate  remediation  for 
vehicle  allocation  was  offered. 

Capability  for  real-time  play  modification  is  a  central  tenet  of  AFDD’s  Delegation  Control  over  less  flexible, 
scripted  vehicle  behaviors.  The  center  portion  of  the  Plays  page  was  dedicated  to  play  editing.  The  operator 
was  able  to  modify  default  assets  assigned  to  a  play  and  waypoint  navigation  in  real-time.  To  further  refine  the 
play,  the  operator  could  change  navigation  parameters  and/or  search  geometry  associated  with  a  waypoint. 
Waypoints  could  be  added  or  deleted  by  the  operator  on  any  play  in  real-time. 

The  lower  portion  of  the  Plays  page  contained  a  Play  Status  window  for  all  plays  uninitiated  and  in  progress. 
Plays  were  listed  in  order  from  highest  to  lowest  priority.  The  operator  had  the  ability  to  increase  or  decrease  a 
play’s  priority  through  selection  of  up  and  down  arrows.  As  additional  plays  were  built,  both  the  listing  of 
plays  and  priority  of  plays  were  updated.  Plays  were  automatically  removed  from  the  status  window  upon 
completion.  Details  of  play  status  included:  priority  order,  assets  assigned,  time  to  start,  and  time  remaining 
on  the  play.  Operator  changes  to  a  play  were  immediately  updated  in  the  play  status  window.  Deletion  of  a 
play  was  accomplished  by  selecting  a  red  A,  thereby  “closing”  the  play. 

The  sensor  display  was  located  on  the  right  side  of  the  Delegation  Control  operator  interface.  The  sensor 
display  was  divided  into  four  sensor  windows.  Sensor  windows  were  outlined  in  the  correlated  color  of  the 
vehicle  icon  offering  visual  cues  to  imagery  source.  In  the  upper  left  corner  of  each  window  a  text  description 
of  the  vehicle  source  was  displayed.  In  further  detail,  the  top  sensor  window  displayed  the  actively  controlled 
sensor.  The  lower  side-by-side  sensor  windows  displayed  a  second  and  third  vehicles’  imagery.  The  bottom 
thumbnail  image  contained  the  fourth  vehicle’s  imagery.  Although  small,  the  fourth  thumbnail  display 
remained  a  live  image  instead  of  a  static  jpeg.  Continual  presence  of  all  four  sensor  displays,  regardless  of 
size,  was  a  design  decision  made  to  support  the  operator’s  situation  awareness  and  serve  as  a  reminder  of 
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controlled  payloads.  Operator  selection  of  a  vehicle  for  active  sensor  control  was  accomplished  by  selecting  a 
vehicle’s  payload  feed  with  the  mouse.  Once  selected,  the  vehicle’s  imagery  replaced  alternate  imagery  in  the 
top  window.  Replaced  imagery  switched  display  locations  with  the  recently  selected  active  vehicle  sensor. 

Superimposed  payload  symbology  displayed  information  related  to  vehicle  heading  and  sensor  direction  on 
the  active  sensor  window.  Specifically,  the  symbology  included  a  screen-fixed,  compressed  heading  tape, 
marked  in  10-deg  increments  and  labeled  in  30-deg  increments.  The  sensor  direction  carrot  (sensor  heading) 
remained  in  the  center  of  the  180-deg  heading  tape  as  it  moved  across  the  top  of  the  display.  A  vehicle  icon, 
identical  to  the  moving  map,  depicted  vehicle  heading  with  a  screen-delimiting  arrow  appearing  when  sensor 
and  vehicle  heading  diverged  more  than  90  deg.  At  such  time,  vehicle  heading  was  displayed  to  the  right  of 
the  arrow  as  a  digital  readout,  as  this  information  was  no  longer  visible  on  the  compressed  tape. 

Superimposed,  geo-referenced  waypoint  markers  representing  vehicle  flight  path  /  ground  track  overlaid 
sensor  imagery.  This  use  of  augmented  reality  was  intended  to  assist  the  operator  in  navigation  and  map-to- 
video  correlation. 

Crosshair  symbology  was  displayed  on  vehicle  sensor  windows  during  plays  involving  weapons  engagement. 
For  example,  during  a  Prosecute  Target  play,  message  prompts  to  ready  and  engage  the  laser  designator  were 
concurrent  with  the  display  of  superimposed  flashing  brackets  around  the  crosshairs.  Once  target  lock-on  was 
accomplished,  brackets  continued  to  be  displayed  without  flashing.  Crosshair  symbology  and  brackets  were 
positioned  only  on  sensor  windows  of  vehicles  assigned  to  the  play  and  disappeared  with  play  termination. 
This  design  decision  supported  operator  SA  of  vehicle  collaboration  in  a  weapons  engagement. 

In  addition  to  menu  selection  by  mouse,  an  operator  could  exercise  voice  commands  to  select  a  play  from  the 
Play  Library.  By  utilizing  voice  commands,  the  operator  was  able  to  rapidly  initiate  a  play  during  time-critical 
mission  phases  (e.g.,  response  to  TIC).  Operators  were  able  to  circumvent  mouse  navigation  of  multi-level 
menus  by  annunciating  an  exact  parameter  to  be  changed.  Use  of  voice  commands  was  incorporated  as  an 
intuitive  supplemental  capability  to  augment  Delegation  Control.  It  was  considered  a  time  saving  measure  and 
assisted  in  the  reduction  of  operator  workload. 


15.5  HUMAN  FACTORS  ISSUES  EXPLORED 

The  purpose  of  this  flight  demonstration  was  to  successfully  demonstrate  delegation  control  of  multiple 
unmanned  air  and  ground  vehicles  by  a  single  operator  in  a  collaborative  urban  scenario  [2].  In  an  extension 
of  empirical  studies  [3], [4], [5]  and  flight  demonstrations,  it  was  hypothesized  that  a  single  operator  could 
effectively  monitor  four  unmanned  heterogeneous  assets  with  reasonable  workload  levels  and  high  situational 
awareness  using  Delegation  Control  employment. 

Specifically,  demonstration  objectives  focused  on  showcasing  solutions  related  to  Human-Machine  Interface 
(HMI)  challenges  involved  with  control  of  multiple  payloads  and  vehicle  platforms.  Best  practice  solutions 
from  human  factors  principles  and  literature  were  applied.  A  Delegation  Control  interface  was  developed  by 
U.S.  Army  AFDD  to  support  control  and  monitoring  of  four  payloads.  In  addition,  voice  recognition  control 
was  implemented  for  multiple  vehicle  control  in  high  workload  mission  segments  (e.g.,  response  to  troops  in 
contract.)  Automation  transparency  was  deemed  significant  to  operator  SA.  Therefore,  textual  presentation  of 
automation  feedback  related  to  simultaneous  play  execution  was  generated  within  a  flight  plan  information 
window.  Overall,  demonstration  success  was  defined  as  the  ability  for  a  single  operator  to  execute  a  mission 
with  multiple  unmanned  assets  in  collaboration  with  tolerable  workload  and  high  situation  awareness. 
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15.6  UNMANNED  SYSTEMS  USED 

Two  live  unmanned  assets  and  two  virtual  unmanned  aerial  vehicles  were  used  in  this  flight  demonstration: 
a  Yamaha  RMAX  helicopter  modified  for  high-level  autonomous  operations  was  flown  in  the  demonstration 
(Figure  15-5);  and  a  Max  5 A  Rover  Unmanned  Ground  Vehicle  (UGV)  manufactured  by  Senseta,  Inc.  and 
operated  by  Carnegie  Mellon  University  Silicon  Valley  (Figure  15-6). 


Figure  15-5:  Live  Yamaha  RMAX  (VTOL  UAV). 


Figure  15-6:  MAX  Rover  UGV. 


A  standard  RMAX  features  a  3  m  diameter  rotor  and  maximum  take-off  weight  of  94  kg.  Additional  payloads 
consisting  of  a  3D  scanning  hemispherical  LADAR  and  autonomous  flight  control  system  mounted  on  the 
aircraft  increased  the  original  baseline  weight  to  172  lbs.  Custom  payload  modifications  were  made  by  U.S. 
Army  AFDD  to  achieve  autonomous  flight  capabilities  to  include  obstacle  navigation.  Flight  controls  and 
navigation  were  all  processed  onboard  utilizing  a  Pentium  III  computer  and  NovAtel  SPAN/LN200  Inertial 
Navigation  Systems.  Mission  duration  for  the  RMAX  was  approximately  60  minutes. 

A  UGV  was  a  4-wheeled,  man-portable  Max  Rover  weighed  approximately  9  kg  and  had  a  cruise  speed  of 
1 1  mph.  The  onboard  computing  platform  consisted  of  a  2.0  GHz  entium-M  with  1  GB  RAM. 

Two  virtual,  fixed  wing  flight  assets  were  included  in  the  flight  demonstration  for  the  purpose  of  showcasing 
collaborative  multi-vehicle  plays.  Nominally,  a  Shadow  tactical  UAS  and  Warrior  Alpha  were  simulated. 
Typical  altitudes,  airspeeds,  and  payload  characteristics  were  emulated  through  the  U.S.  Army  AFDD’s  Multi- 
UAS  Simulation  (MUSIM)  software. 

15.6.1  Sensor  Payloads/Imagery 

The  Yamaha  RMAX  was  equipped  with  a  fixed  forward  electro-optical,  color  camera  with  ability  to  pitch 
+10/-100  deg.  640  x  480  grayscale  images  were  telemetered  over  Wi-Fi  at  a  quality  of  25%  and  updated  at 
8  Hz. 

The  sensor  platform  on  the  MAX  Rover  consisted  of  a  Videre  synchronized  stereo  pair  system  camera. 
Camera  manipulation  was  accomplished  in  pitch  (-30/+80  deg)  and  pan  (+/-170  deg)  axes.  A  series  of  jpeg 
images  was  transmitted  and  updated  at  15  Hz  to  the  Delegation  Control  operator  display.  Imagery  capture  and 
resolution  was  320  x  240. 
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The  virtual  Shadow  UAS  was  nominally  equipped  with  Electro-Optical/Infrared  (EO/IR)  sensor  and  laser 
designator.  The  virtual  Warrior  Alpha  was  nominally  equipped  with  EO/IR  and  four  Hellfire  missiles.  For  the 
purpose  of  demonstration,  virtual  sensors  had  360  deg  pan  capability  and  +45/-110  deg  pitch  limits.  Sensor 
slew  rate  was  set  at  60  deg/second.  Zoom  capabilities  supported  a  progressive  change  in  FOV  from  2  to 
16  deg  (x  -  8x).  Precise  modeling  of  the  laser  designator  and  weapons  delivery  system  was  not  considered 
necessary  for  this  demonstration.  Thus,  a  low  fidelity  emulation  of  the  weapons  systems  was  employed. 

15.6.2  Telemetry 

The  mobile  ground  station  was  equipped  with  two  antennas  for  900  MHz  and  2.4  GHz  telemetry.  Payload 
telemetry  for  the  RMAX  used  the  902  -  928  MHz  frequency  range.  The  72.11  and  72.13  MHz  frequencies 
were  used  for  backup  aircraft  control.  An  omni-directional  antenna  for  datalink  transmissions  to  the  RMAX 
was  mounted  on  the  tallest  building  at  the  MOUT  site,  allowing  for  complete  site  coverage.  Adjacent  to  the 
RMAX  antenna,  a  manually  swiveled  Yagi-Uda  antenna  was  utilized  for  Wi-Fi  datalink  transmission  to  the 
MAX  Rover. 


15.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

The  NATO  working  group  served  as  an  informal  crew-station  working  group.  The  demonstration  plan,  plays 
and  data  collection  were  presented,  discussed  and  vetted  at  numerous  meetings  prior  to  the  demo.  The  USAF 
integrated  and  provided  technical  expertise  for  the  voice  recognition  system. 


Planning/Design 

Execution 

Analysis 

Communication 

All 

All 

All 

Coordination 

USAF,  Canada, 
Germany 

Collaboration 

None 

15.8  SUMMARY  OF  TD  RESULTS 

The  TD  demonstrated  simultaneous  control  of  multiple  vehicles  by  a  single  operator.  The  workload  was 
acceptable  and  situation  awareness  reasonable.  These  were  actual  vehicles  in-flight  and  on  the  ground  as  well 
as  virtual  aircraft.  This  was  accomplished  via  Playbook  (DELCON)  methodology.  Voice  recognition  to  “call” 
the  plays  was  also  demonstrated. 


15.9  LESSONS  LEARNED 

Under  these  circumstances,  a  single  UAS  operator  can  simultaneously  control  multiple  UAS.  Voice  recognition 
can  be  integrated  seamlessly  into  this  environment. 
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15.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  study  was  limited  primarily  in  that  it  was  concept  demonstration  not  a  flight  test.  That  is,  it  was  not  an 
actual  experiment,  it  was  demonstrating  concepts  previously  tested  in  the  laboratory.  Further,  no  actual  UAS 
operators  controlled  the  system,  they  were  controlled  by  one  of  the  experimenters.  Additional  limitations 
included: 

•  Unfavorable  weather  conditions  for  flight  testing  of  the  RMAX  proved  to  be  one  of  the  largest 
obstacles  faced  by  the  research  team.  This  is  being  addressed  in  future  demonstrations/flight  tests 
through  selection  of  a  more  suitable  location  (with  more  favorable  weather  conditions),  as  well  as 
investigation  of  a  weather  impacts  and  route  planning  tool  developed  by  ARL,  called  MyWIDIA 
(My  Weather  Impacts  Decision  Aid). 

•  Inability  to  integrate  differential  GPS  onto  the  MAX  Rover  within  the  allotted  timeframe,  combined 
with  limited  GPS  reception  and  safety  concerns  regarding  road  widths  at  the  Ft.  Ord  MOUT  site 
prevented  the  research  team  from  allowing  MUSIM  to  send  automatic  waypoints  to  the  vehicle, 
despite  having  developed  and  tested  this  capability  beforehand.  Again,  this  limitation  could  be 
overcome  by  using  a  more  suitable  location,  e.g.,  one  with  larger  areas  for  transit  to  offset  location 
errors,  or  one  with  better  GPS  reception. 

•  Lack  of  access  to  trained  operators  significantly  hindered  the  research  team’s  ability  to  collect  any 
human  performance  data.  This  is  being  addressed  in  a  future  flight  test  by  using  general  aviation 
pilots  that  have  been  trained  on  the  MUSIM  system  and  participated  in  simulation  experiments  in 
AFDD’s  laboratory. 


15.11  CONCLUSIONS 

This  demonstration  was  deemed  a  success  for  aptly  showcasing  Delegation  Control  of  multiple  unmanned 
systems  by  a  single  operator.  Navigation  and  payload  control  of  four  unmanned  systems  was  successfully 
monitored  by  a  single  operator  in  a  collaborative  urban  mission  scenario.  Delegation  Control  employment 
strategy  and  interface  design  supported  the  build,  initiation,  modification  and  monitoring  of  simultaneous 
plays  in  progress.  Use  of  voice  recognition  was  considered  an  advantage  to  the  operator  during  time  critical 
mission  phases.  The  operator’s  ability  to  bypass  menus  in  favor  of  voice  recognition  control,  significantly 
decreased  reaction  time  to  external  mission  events.  Dynamic  route  re-  planning  was  effectively  accomplished 
while  plays  were  in  progress.  In  addition,  play  status  was  efficiently  depicted  and  real-time  updates  were 
accomplished  when  play  modification  and  play  terminations  occurred.  Automation  transparency  was 
increased  through  messages  generated  by  the  MUSIM  software  that  described  impacts  of  conflicting  plays. 
Lastly,  the  Play  Status  window  was  considered  a  significant  contribution  to  operator  SA  for  rapid,  at-a-glance 
awareness  of  asset  allocation  and  play  scheduling. 


15.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

This  demonstration  served  to  test  and  extend  Delegation  Control  concepts  initially  sourced  from  empirical 
studies  and  Subject-Matter  Expert  (SME)  interviews.  Operational  limitations  of  Delegation  Control  were 
explored  during  the  demonstration  without  benefit  of  formal  data  collection  or  control  comparison.  Future 
research  will  include  flight-tested  data  to  support  Delegation  Control  employment  strategies  over  alternative 
control  strategies.  Performance  data  and  subjective  ratings  of  operator  workload  and  SA  will  be  collected  in 
future  flight  tests.  To  ensure  operationally  relevant  and  tactically  valid  play  definitions,  U.S.  Army  AFDD  has 
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plans  to  vet  and  expand  play  definitions  with  the  user  community.  Interviews  with  returning  warfighters  will 
be  conducted  and  subjective  ratings  collected  on  play  usefulness,  feasibility,  and  projected  frequency  of  use. 
Modifications  to  the  play  library  and  play  defaults  will  be  made  where  appropriate.  U.S.  Army  AFDD  will 
continue  to  explore  Delegation  Control  as  an  effective  means  for  multi-vehicle  unmanned  systems  control  by 
a  single  operator.  AFDD  plans  to  continue  testing  Delegation  Control  in  lab  studies  and  flight  tests.  Near-term 
plans  include  the  testing  of  the  top  five  user-rated  plays,  expanding  play  definitions,  and  providing  intelligent 
automation  feedback  when  plays  have  been  degraded.  Possibilities  for  play  rehearsal  will  also  be  explored. 
In  the  out-  years,  plans  exist  to  implement  Delegation  Control  from  a  manned  platform  in  support  of  manned- 
unmanned  teaming. 
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16.1  DATES 

2009-2011. 

16.2  LOCATION 

Aberdeen  Proving  Ground,  MD,  USA. 


16.3  SCENARIO/TASKS 

Urban  reconnaissance. 


16.4  TECHNOLOGIES  EXPLORED 

Intelligent  agent  to  coordinate  a  team  of  ground  robots  for  urban  reconnaissance  tasks. 


16.5  HUMAN  FACTORS  ISSUES  EXPLORED 
16.5.1  Introduction 

Future  military  operations  will  be  complex  and  diverse.  U.S.  forces  and  its  allies  will  be  engaged  simultaneously 
on  a  number  of  fronts  under  different  combat  conditions  while  fighting  adversaries  using  a  variety  of  tactics. 
Because  of  the  changing  demographics,  many  of  our  engagements  will  be  in  urban  areas  with  entrenched 
adversaries  who  do  not  have  to  defeat  us;  instead  they  need  only  to  out- wait  us  [1].  These  conflicts  will  require 
the  Army  to  put  Soldiers  “in  harm’s  way”  and  in  the  process  encourage  adversaries  to  use  any  means  at  their 
disposal  to  outlast  our  resolve.  One  possible  solution  will  be  the  implementation  of  robotic  systems  that  can 
replace  Soldiers  on  the  battlefield  increasing  our  forces’  survivability  and  durability.  Eventually  these  systems 
will  be  able  to  operate  24/7  in  difficult  terrain  testing  the  Soldier  operator’s  ability  to  supervise  these  assets  while 
conducting  normal  operations.  Also,  the  possibility  of  a  robotic  battlefield  creates  a  number  of  human  factors  as 
well  as  ethical  issues  related  to  non-human  intelligence  conducting  combat  missions  [2], [3].  One  obvious  issue  is 
that  the  proliferation  of  intelligent  systems  could  easily  overwhelm  the  current  force’s  ability  to  adequately 
supervise  these  systems.  Hundreds  of  Uninhabited  Vehicles  (UVs),  both  aerial  and  ground,  will  share  the 
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battlefield  with  hundreds  of  manned  systems  and  conduct  numerous  missions  concurrently.  In  these  situations, 
the  military  will  not  be  able  to  afford  the  manpower  to  control  individual  systems;  instead,  future  missions  will 
require  single  operators  to  supervise  multiple  systems.  This,  in  turn,  will  necessitate  some  degree  of  UV 
autonomy.  Unfortunately,  increases  in  autonomy  will  present  its  own  set  of  problems,  including  tunnel  vision, 
misuse  and  disuse  of  automated  systems,  complacency,  and  loss  of  situation  awareness  [4], [5].  More  germane  to 
our  current  discussion,  increases  in  autonomy  will  not  overcome  the  human’s  span  of  apprehension  limits  related 
to  monitoring  multiple  systems  at  the  same  time  [6], [7]. 

Researchers  have  proposed  a  number  of  solutions  to  the  potential  issues  of  a  robotic  battlefield,  such  as  setting 
up  a  UV  call  center  in  which  robots  will  query  the  human  operator  only  when  there  is  a  problem.  The  operator 
will  make  the  necessary  adjustments  but  will  not  be  required  to  monitor  robots  continuously  [8].  The  obvious 
problem  with  this  solution  is  that  it  assumes  that  the  UV  will  be  able  to  self-diagnose  its  own  problems; 
additionally,  the  number  of  operator-robot  interactions  is  expected  to  increase  exponentially  during  the  heat  of 
combat,  making  the  call  center  ineffective  during  the  most  critical  time  periods.  As  a  potential  safeguard,  a 
number  of  researchers  have  suggested  algorithms  that  share  control  responsibility  among  robots  and  humans 
as  a  function  of  either  the  robots’  behavior  or  the  operator’s  cognitive  state.  Closely  aligned  concepts  involve 
play-book  solutions  that  permit  the  operator  to  insert  pre-programmed  algorithmic  solutions  that  control 
robots  during  difficult  mission  segments  [9].  This  generic  class  of  adaptive  systems  is  designed  to  keep 
operators  in  the  decision  loop  while  keeping  the  overall  supervisory  burden  within  efficient  cognitive  limits 
[9]-[12].  However,  while  this  approach  mitigates  problems  during  high  workload,  it  does  not  overcome 
cognitive  limitations  when  the  number  of  human-robot  interactions  surpasses  human  cognitive  capacity  [6]. 
We  will  examine  two  approaches  that  directly  address  the  many-to-one  problem: 

a)  Distributed  intelligence  using  swarm  technologies;  and 

b)  Centralized  intelligence  using  an  intelligent  agent  as  an  intermediate  supervisor. 

The  Tech  Demo  (RoboLeader)  demonstrates  the  dynamics  of  an  intelligent  agent  interacting  with  the  human 
operators  to  coordinate  a  team  of  ground  robots  conducting  an  urban  reconnaissance  mission.  In  artificial 
intelligence,  an  intelligent  agent  is  typically  defined  as  “an  autonomous  entity  which  observes  and  acts  upon 
an  environment  and  directs  its  activity  towards  achieving  goals”  [13].  This  definition  covers  a  variety  of 
possible  uses  for  intelligent  agents,  from  swarms  with  individual  agents  of  limited  intelligence  that  evince 
sophisticated  behaviours  holistically,  to  agents  that  respond  to  particular  tasks  in  a  manner  that  emulates 
human  intelligence.  As  a  contrast  to  RoboLeader’ s  theoretical  underpinnings,  we  will  initially  discuss  swarms, 
emphasizing  recent  research  from  the  Army  Research  Laboratory  (ARL;  Figure  16-1).  The  purpose  of 
discussing  swarms  will  be  to  delimit  the  theoretical  possibilities  for  supervising  multiple  UVs  in  order  to  set 
the  stage  for  our  discussion  of  RoboLeader. 
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Figure  16-1:  Swarm  Display  [16]. 


Swarm  technologies  are  a  special  type  of  distributed  intelligence  wherein  each  component  has  a  limited 
capacity  to  respond  to  its  environment  and  the  intelligence  resides  within  the  combined  behavior  of  the  group. 
Modeling  swarms  is  an  example  of  bio-inspired  engineering  using  techniques  that  mimic  collective  behaviors 
of  organisms  such  as  of  birds,  ants  and  bees.  Scientists  have  modeled  swarm  behavior  making  simple 
assumptions  about  the  rules  that  individual  members  of  the  swarm  use  to  permit  the  group  to  obtain  its  desired 
end  states.  Rules  can  be  such  simple  actions  as:  stay  close  to  your  neighbor,  avoid  collisions,  and  move  in  the 
same  direction.  For  example,  Craig  Reynolds  developed  an  algorithm  describing  flocking  behavior,  including 
obstacle  avoidance,  to  mimic  avian  flight  paths  accurately  [14].  His  Boids  program  was  so  successful  that  it 
was  used  to  develop  flight  animations  in  the  Bat  Man  Returns  movie  in  1992.  A  curious  aspect  of  swarm 
behavior  is  that  no  single  agent  is  in  charge  making  the  Swarm  invulnerable  to  an  individual’s  poor  decisions. 
For  example,  in  the  biological  domain,  scout  ants  will  go  off  in  many  directions  and  foragers  will  count  those 
coming  back  in  a  particular  direction  to  decide  in  which  direction  to  find  food  for  the  colony  [15]. 

More  pertinent  to  military  problems  is  the  work  that  ARL  researchers  have  conducted  using  swarm  methods 
to  control  sentry  robots  for  convoy  protection  (Figure  16-1)  [16].  The  robots  use  rules  similar  to  the  ones 
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instantiated  for  flocking  behaviors  to  form  an  ellipse  around  the  convoy,  with  individual  robots  venturing 
neither  too  close  nor  too  far  from  the  convoy.  The  algorithm  had  to  be  robust  enough  to  account  for 
unexpected  weather,  obstacles,  etc.,  in  the  convoy’s  path.  Using  a  simulation  program  with  artificial  obstacles, 
the  robots  were  able  to  stay  within  the  prescribed  distance  of  the  convoy  85%  of  the  time.  The  problem  with 
swarm  technology  is  that  swarms  are  most  useful  for  simple  problems  wherein  adaptation  to  novel  conditions 
is  not  an  important  part  of  the  problem  set.  Also,  swarms  are  difficult  to  control  because  they  are  essentially 
interacting  components  with  no  clear  means  of  supervisory  control  [17]. 

For  more  complex  problems,  hierarchical  agent  systems  are  being  designed  with  agents  that  have  specialized 
intelligence  embedded  within  multi-layer  architectures.  The  individual  agents  have  specific  tasks  and  a  means 
of  communicating  with  other  agents.  The  ability  to  divide  task  complexity  between  senior  (more  capable) 
agents  and  specialized  (less  capable)  ones  allows  hierarchies  to  adjust  to  greater  complexity  and  to  better 
adjust  to  change.  The  disadvantages  are  that  as  hierarchies  become  more  complex,  the  algorithms  to  control 
them  also  become  more  cumbersome,  and  even  then,  agent  hierarchies  are  still  challenged  by  truly  novel 
situations.  Furthermore,  the  more  levels  involved  and  the  more  entities  that  need  to  be  controlled,  the  more 
likely  it  is  that  communications  among  agents  will  become  a  serious  problem.  Carnegie  Mellon  University 
(CMU)  developed  algorithms  using  various  bidding  techniques  in  order  to  assign  agents  specific  tasks. 
Bidding  for  tasks  in  the  same  neighbourhood  reduced  the  size  of  auctions  and  it  also  reduced  the  distance  that 
each  agent  was  required  to  travel  to  complete  its  mission.  In  general,  three  levels  of  hierarchy  were  found  to 
be  the  most  efficient  network  size  in  order  to  trade-off  number  of  agent  specialties  with  network  complexity 
[18]-[20]. 

Agent  technologies  with  military  import  have  been  demonstrated  for  a  number  of  realistic  applications. 
For  example,  the  L-3  Corporation  and  CMU  used  agents  to  successfully  control  multiple  Unmanned  Aerial 
Vehicles  (UAV)  with  Received  Signal  Indicator  (RSI)  sensors  to  locate  targets  cooperatively  during  high 
fidelity  simulations  [21].  Researchers  for  the  Canadian  Defence  Research  and  Development  Laboratories 
(DRDL)  demonstrated  the  utility  of  agent  hierarchy  technology  for  UAV  control  in  a  more  complex 
simulation  environment  [22].  Hou  and  his  colleagues  [22]  used  senior  agents  directing  working  agents  who  in 
turn  directed  junior  agents.  Their  simulation  demonstrated  agents  in  concert  with  the  operator,  planning 
multiple  UAV  missions,  navigating  UAVs  to  the  target  area,  and  upon  arrival  directing  sensors  to  locate  the 
targets.  Similar  agent  technology  has  been  used  for  cooperative  control  of  multiple  UAVs  at  the  German 
Bundeswehr  University  in  Munich  using  live  UAV  demonstrators  [23]. 

16.5.2  Human- Agent  Teaming 

An  important  feature  of  most  military-related  agent  research  is  that  agents  are  imbued  with  limited  autonomy. 
Agents  can  perform  specialized  functions  but  authority  resides  with  a  human  supervisor  for  safety  and  tactical 
reasons  [2], [23].  Human-agent  teams  seem  to  be  particularly  effective  for  open  ended  missions  in  which  not 
all  events  can  be  pre-programmed  (i.e.,  most  combat  situations).  Successful  agent  technologies  take  advantage 
of  the  differences  between  human  and  agent  strengths.  Human  reasoning  has  very  different  characteristics 
than  algorithmic  reasoning.  Johnson-Laird  [24]  points  out  that  humans  are  rational  but  do  not  use  formal  logic 
in  everyday  decision  making.  They  tend  to  structure  problems  by  focusing  on  only  some  of  the  possible 
logical  implications;  permitting  humans  to  rapidly  but  sometimes  erroneously  solve  real-world  problems 
(see  for  example,  Monty  Hall  problem  [25]).  On  the  other  hand,  using  inductive  processes,  humans  are  able  to 
visualize  numerous  possible  patterns  that  would  overwhelm  current  agent  technology.  Also,  we  tend  to 
overlook  the  advantages  of  consciousness  which  gives  humans  an  acute  awareness  of  the  present  and  an 
intuitive  feel  for  future  states  as  well  as  a  sense  of  purpose  [26].  Human  intuition  is  not  an  inexplicable 
process  but  rather  it  involves  matching  multiple  memory  traces  with  current  cues  that  alert  Soldiers  to  possible 
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incongruities  in  the  environment.  Experts,  in  particular,  are  able  to  pick  up  on  cues  that  allow  them  to 
circumvent  detailed  analysis  [27].  Also,  humans  can  understand  and  react  to  situations  in  terms  of  overall 
intent  rather  than  specified  objectives.  Artificial  agents  are  able  to  supplement  human  intelligence  by  their  use 
of  more  formal  logic  and  their  use  of  complex  optimization  algorithms  to  solve  circumscribed  problem  sets. 
Agents  also  reduce  workload  by  being  able  to  attend  to  multiple  functions  that  would  overwhelm  Soldiers 
during  the  heat  of  combat.  Human-agent  teams  are  especially  important  in  military  environments  because  no 
set  of  agents  in  the  foreseeable  future  will  be  able  to  understand  the  nuanced  political  and  ethical  implications 
facing  U.S.  ground  troops  [2], [28]. 

The  purpose  of  the  below  research  is  to  better  understand  how  humans  and  intelligent  agents  work  together 
effectively  as  team  to  engage  in  military  missions  wherein  Soldiers  will  be  required  to  supervise  multiple  UVs 
during  high  workload  combat  missions  [4], [29].  RoboLeader  is  a  simple  hierarchical  system  consisting  of 
human  operators,  intelligent  agents  that  can  communicate  with  both  the  human  supervisor  and  other  agents, 
and  less  intelligent  agents  consisting  of  multiple  UVs  (robots)  who  conduct  prescribed  missions. 
The  dynamics  of  the  interactions  and  report  of  our  initial  findings  will  constitute  the  demonstrations  and  are 
described  below. 

16.5.3  Framework  Applied  to  HFM-170 

The  purpose  of  our  demonstration  is  to  show  the  advantages  of  a  hybrid  supervisory  system  with  a  centralized 
agent  controlling  multiple  UVs  in  an  urban  combat  environment.  RoboLeader  will  be  contrasted  with  swarm 
systems  and  we  will  discuss  research  showing  its  advantages  in  controlling  up  to  8  robots.  Past  research 
indicates  that  autonomous  cooperation  between  robots  can  improve  the  performance  of  the  human  operators 
[30],  as  well  as  enhancing  overall  human-robot  team  performance  [31].  The  current  research  paradigm 
addresses  the  control  structure  and  interface  requirements  between  the  supervisory  human  operator  and 
robotic  agents  as  number  of  agents,  workload,  target  mobility,  and  reliability  level  vary  systemically  [32], [33]. 


16.6  UNMANNED  SYSTEMS  USED 

Simulated  unmanned  ground  vehicles. 


16.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

Communications  included  the  HFM-170  community  by  sharing  papers  and  presentations.  Direct  interactions 
were  with  TNO  Netherlands,  Air  Force  Research  Laboratory  (AFRL)  and  DRDC  Canada.  Interactions  with 
Canada  include  sharing  software  and  concrete  plans  for  joint  research  on  intelligent  agents,  TNO  participated 
in  field  experiments  at  Ft.  Benning  and  AFRL  contributed  to  our  voice  research. 
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ORGANIZATION 


16.8  SUMMARY  OF  TD  RESULTS 

16.8.1  Experiment  1 

We  investigated  the  effectiveness  of  RoboLeader,  an  intelligent  agent  that  could  help  the  human  operator 
control  a  team  of  robots,  for  enhancing  the  overall  human-robot  teaming  performance.  We  compared  the 
operators’  target  detection  performance  in  the  4-robot  and  8 -robot  conditions  [32].  The  Mixed  Initiative 
Experimental  (MIX)  Testbed  was  modified  and  used  as  the  simulator  [34].  The  MIX  Testbed  is  a  distributed 
simulation  environment  for  investigation  into  how  unmanned  systems  are  used  and  how  automation  affects 
performance.  The  Operator  Control  Unit  (OCU)  of  the  MIX  Testbed  was  modeled  after  the  Tactical  Control 
Unit  developed  under  the  ARL  Robotics  Collaborative  Technology  Alliance  (Figure  16-2).  This  platform 
includes  a  camera  payload,  and  supports  multiple  levels  of  automation.  Users  can  send  mission  plans  or 
teleoperate  the  platform  with  a  joystick  while  receiving  video  feed  from  the  camera  payload.  Typical  tasks 
include  reconnaissance  and  surveillance. 


Figure  16-2:  RoboLeader  User  Interface. 


Participants  were  randomly  assigned  to  the  RoboLeader  group  or  the  Baseline  (no  RoboLeader)  group  before 
their  sessions  started.  Each  experimental  session  had  two  scenarios,  each  lasting  approximately  30  minutes, 
in  which  participants  used  their  robotic  assets  to  locate  20  targets  (i.e.,  10  insurgents  carrying  weapons  and 
10  Improvised  Explosive  Devices  [IEDs])  in  the  remote  environment.  There  were  4  robots  available  in  one 
scenario  and  8  in  the  other.  The  order  of  scenarios  was  counterbalanced  across  participants.  When  each 
scenario  started,  the  robots  began  by  following  pre-planned  routes  at  which  time  the  operator’s  task  of 
monitoring  the  environment  and  detecting  insurgents/IEDs  began.  The  robots  did  not  have  Aided  Target 
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Recognition  capability;  therefore,  the  participants  had  to  detect  the  10  insurgents  and  10  IEDs  by  themselves. 
There  were  friendly  dismounted  soldiers  and  civilians  in  the  simulated  environment  to  increase  the  visual 
noise  for  the  target  detection  tasks.  The  participants  were  told  that  their  objective  was  to  finish  reconnoitering 
the  area  using  their  robotic  assets  in  the  least  amount  of  time  possible.  Therefore,  when  re-planning  a  route, 
the  participant  and/or  RoboLeader  were  required  to  consider  both  the  effectiveness  and  efficiency  of  the  new 
route.  In  each  scenario,  there  were  six  events  that  required  revisions  to  a  robot’s  current  plans/route.  Once  an 
event  transpired,  the  baseline  participants  had  to  notice  that  the  event  had  occurred,  and  then  they  would 
re-route  the  robot  that  was  affected  by  the  event.  For  those  in  the  RoboLeader  condition,  the  RoboLeader 
recommended  plan  revisions  to  the  operator,  who  could  either  accept  the  plans  or  modify  them  as  necessary. 
In  each  scenario,  there  were  5  Situation  Awareness  (SA)  queries,  which  were  triggered  based  on  time 
progression  (e.g.,  3  minutes  into  the  scenario).  The  SA  queries  included  questions  such  as  “which  areas  have 
the  robots  searched?”  (participants  were  instructed  to  mark  the  searched  areas  on  a  blank  map),  “which  of 
your  robots  is  the  closest  to  [Area  of  Interest]”,  etc.  The  OCU  screen  was  blank  when  an  SA  query  was 
triggered,  and  only  the  SA  query  and  the  answer  box  were  displayed  on  the  screen. 

The  study  was  a  mixed  design,  with  RoboLeader  (with  or  without  RoboLeader  [Baseline])  as  the  between- 
subject  variable,  and  the  number  of  Robots  used  in  the  scenario  (4  vs.  8)  as  the  within- subject  variable. 
Dependent  measures  included  number  of  targets  located  and  identified,  the  operator’s  SA  of  the  mission 
environment  as  well  as  awareness  of  the  status  of  the  individual  robots,  and  the  operator’s  self-assessed 
workload.  A  mixed-design  analysis  of  covariance  with  RoboLeader  (with  or  without  RoboLeader)  as  the 
between- subject  factor  and  number  of  Robots  (4  vs.  8)  as  the  within-subject  factor  was  used  to  evaluate  the 
operator’s  performance  differences  among  the  four  conditions.  Participants’  spatial  ability  (composite  score  of 
two  spatial  tests)  and  their  attentional  control  survey  scores  were  used  as  covariates. 

Results  showed  that  participants  detected  significantly  fewer  targets  and  had  significantly  worse  SA  when 
there  were  8  robots  compared  to  the  4-robot  condition.  Those  participants  with  higher  spatial  ability  detected 
more  targets  than  did  those  with  lower  spatial  ability.  Participants’  self-assessed  workload  was  affected  by  the 
number  of  robots  under  control,  their  gender,  and  their  attentional  control  ability.  Although  there  was  no 
significant  difference  in  overall  target  identification  between  RoboLeader  and  baseline  conditions,  there  was  a 
12%  reduction  in  mission  completion  time  for  the  RoboLeader  condition. 

16.8.2  Experiment  2 

In  the  first  experiment,  the  simulated  reliability  level  of  RoboLeader  was  100%  (i.e.,  no  false  alarms  or  misses). 
In  Experiment  2,  the  effects  of  various  reliability  levels  for  RoboLeader  on  operator  performance  were 
investigated  [32].  The  participants’  task,  as  in  Experiment  1,  was  to  manage  four  robots  with  the  assistance  of 
RoboLeader  while  searching  for  hostile  targets  via  streaming  video  from  the  robots.  The  reliability  of 
RoboLeader’s  solutions  was  manipulated  to  be  either  False-Alarm  Prone  (FAP)  or  Miss  Prone  (MP),  with  a 
reliability  level  of  either  60%  or  90%.  Furthermore,  Experiment  2  simulated  a  multi-tasking  environment  rather 
than  a  dual-tasking  environment  as  in  Experiment  1.  In  addition  to  the  target  detection  and  route  revision  tasks, 
the  participants  had  to  simultaneously  perform  a  gauge  monitoring  task  and  a  communication  task.  Finally,  the 
visual  density  of  the  simulated  environment  was  manipulated;  there  were  twice  as  many  entities  in  the  high 
density  environment  as  in  the  low  density  environment.  The  experiment  is  a  mixed  design,  with  RoboLeader 
Imperfection  Type  (FAP  vs.  MP)  and  Reliability  Level  (60%  vs.  90%)  as  the  between-subject  factors  and  Visual 
Density  (High  vs.  Low)  of  the  simulated  environment  as  the  within-subject  factor. 

Participants  were  randomly  assigned  to  the  FAP60,  FAP90,  MP60,  or  MP90  group  (with  10  participants  per 
group)  before  test  sessions  started.  The  participants  were  informed  that  RoboLeader  was  either  FAP  or  MP 
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and  “fairly  but  not  always  reliable”  (for  the  90%  conditions)  or  “not  always  reliable”  (for  the  60%  conditions). 
In  the  MP  scenarios,  participants  were  required  to  notice  and  manually  edit  several  routes  without  the  help  of 
RoboLeader.  RoboLeader’ s  messages  were  displayed  in  the  upper  left  comer  (the  blue  area)  of  the  OCU 
(see  Figure  16-3).  As  in  Experiment  1,  participants  were  told  that  their  objective  was  to  finish  reconnoitering 
the  area  using  their  robots  in  the  least  amount  of  time  possible  while  keeping  all  route  edits  as  close  as 
possible  to  the  original  routes.  Therefore,  when  re-planning  a  route,  the  participants  and  RoboLeader  had  to 
consider  the  effectiveness  and  efficiency  of  a  new  route. 


Figure  16-3:  RoboLeader  in  a  Multi-Tasking  Environment. 

In  the  FAP60  scenario,  there  were  five  tme  events  that  required  revisions  to  a  robot’s  route  and  four  FAs  that 
RoboLeader  attempted  to  edit  around  when  no  events  occurred.  Participants  could  verify  the  validity  of  the 
RoboLeader  recommendations  by  reviewing  the  map.  A  tme  event  was  associated  with  an  icon  (a  red  square 
for  a  Hostile  Area  and  a  blue  square  for  a  High  Priority  Area,  see  Figure  16-3),  but  FAs  were  not.  In  the 
FAP90  scenario,  there  were  five  tme  events  that  required  revisions  to  a  robot’s  route,  and  one  FA.  In  the 
MP60  scenario,  ten  tme  events  occurred  that  required  revisions  to  a  robot’s  route,  though  RoboLeader  only 
provided  solutions  for  two  of  them.  In  the  MP90  scenario,  ten  tme  events  occurred  and  RoboLeader  provided 
solutions  for  eight  of  them. 

In  addition  to  the  tasks  described  above,  the  participants  simultaneously  performed  a  gauge  monitoring  task  and 
an  auditory  communications  task.  The  gauge  monitoring  task  (upper  left  comer  of  the  OCU)  displayed  four 
gauges  constantly  in  motion  that  entered  an  upper  or  lower  limit  at  various  pre-specified  times  throughout  the 
scenarios.  The  participants  were  required  to  monitor  the  gauges  and  press  a  “Reset”  button  when  any  gauge 
entered  the  upper  or  lower  limit  to  put  the  gauges  back  to  their  normal  levels.  The  auditory  communications  task 
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presented  pre-recorded  questions  at  30  sec  intervals  during  the  scenarios.  The  questions  included  simple 
military-related  reasoning  and  memory  tests.  Participants  used  a  keyboard  to  enter  their  responses  for  the 
questions  into  the  communications  panel  on  the  OCU  (adjacent  to  the  gauges,  see  Figure  16-3). 

Dependent  measures  included  the  number  of  targets  located  and  identified,  the  number  of  routes  successfully 
edited,  the  operators’  SA  of  the  mission  environment,  their  concurrent  task  performance  (gauge  monitoring 
and  auditory  communications)  and  their  perceived  workload.  A  mixed  design  ANCOVA  with  Unreliability 
Type  (FAP  vs.  MP)  and  Reliability  Level  (60%  [Low]  vs.  90%  [High])  as  the  between-subject  factors  and 
Visual  Density  (High  vs.  Low)  as  the  within-subject  factor  is  used  to  evaluate  the  operators’  performance 
differences  among  the  four  conditions.  Participants’  spatial  ability  (composite  score  of  two  spatial  tests)  and 
their  attentional  control  survey  scores  were  used  as  covariates. 

Results  showed  that  the  type  of  RoboLeader  unreliability  (FAP  vs.  MP)  affected  operator’s  performance  of 
visual  scanning  tasks  (target  detection,  route  editing,  and  situation  awareness).  There  was  a  consistent  effect 
of  visual  density  for  multiple  performance  measures.  Participants  with  higher  spatial  ability  performed  better 
on  the  two  tasks  that  required  the  most  visual  scanning  (i.e.,  target  detection  and  route  editing).  Participants’ 
self-assessed  attentional  control  was  found  to  impact  their  overall  multi-tasking  performance,  especially 
during  their  execution  of  secondary  tasks  (communication  and  gauge  monitoring).  The  most  important  finding 
was  the  target  identification  superiority  for  the  FAP  condition  compared  to  the  MP  condition.  This  was  most 
likely  caused  by  the  visual  accessibility  of  the  route  map  making  FA  verification  relatively  easy,  whereas  the 
lack  of  alerts  in  the  MP  condition  required  participants  to  scan  the  map  constantly  thus  missing  the  targets  on 
the  live  video.  This  was  reinforced  by  the  finding  that  MP  conditions  resulted  in  better  overall  SA  which  is 
consistent  with  increased  scanning. 

16.8.3  Experiment  3 

In  2010,  the  capabilities  of  RoboLeader  were  expanded  to  deal  more  specifically  with  dynamic  re-tasking 
requirements  for  persistent  surveillance  of  a  simulated  urban  environment  based  on  various  battlefield 
developments  as  well  as  coordination  between  Unmanned  Aerial  Systems  (UASs)  and  Unmanned  Ground 
Vehicles  (UGVs)  in  pursuit  of  moving  targets  in  urban  environments  (Figure  16-4).  In  Experiment  3, 
we  manipulated  the  level  of  autonomy  of  RoboLeader  and  examined  its  effect  on  the  operator’s  performance 
(i.e.,  plan  revisions  for  the  robots,  the  concurrent  target  detection  task,  and  SA  of  the  mission  environment)  and 
workload  [33].  The  four  levels  of  manipulation  were  as  follows:  Manual  (no  RoboLeader),  Semi- Autonomous 
without  Visualization,  Semi-Autonomous  with  Visualization,  and  Fully  Automated.  The  Semi-Autonomous 
condition  was  divided  into  two  conditions  so  that  the  effect  of  the  visualization  tool  could  be  evaluated. 
The  visualization  tool  informed  the  participant  of  the  synchronization  of  the  robots  as  well  as  overall  entrapment 
effectiveness  of  the  target  based  on  the  movement  of  the  target. 
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Figure  16-4:  RoboLeader  User  Interface  in  Experiment  3. 

During  the  scenarios,  participants  used  their  four  robotic  assets  to  pursue  a  primary  moving  target  (a  truck 
traveling  at  about  3  MPH)  while  monitoring  the  streaming  video  from  the  robots  in  order  to  find  additional 
(secondary)  targets  (insurgents  carrying  weapons)  in  the  mission  environment.  When  the  scenario  for  the 
Manual  condition  started,  the  participants  entered  waypoints  for  each  UGV  manually  and  adjusted  the 
waypoints  based  on  the  movement  of  the  primary  target.  In  the  Semi-Autonomous  conditions,  the  participant 
selected  an  end  point/location  for  the  UGV  at  which  time  RoboLeader  provided  an  optimum  solution  for 
reaching  the  desired  destination.  In  the  visualization  condition,  the  user  could  consult  the  bar  graphs  as  an 
indicator  of  whether  their  point  selections  were  effective  in  terms  of  synchronization  of  the  robots  and 
entrapment  of  the  target  or  if  the  plans  needed  revisions.  The  scores  displayed  in  the  visualization  area  were 
calculated  based  on  the  RoboLeader’ s  encapsulation  algorithm.  Without  visualization,  the  participant  had  to 
determine  if  they  were  properly  cornering  the  target  for  capture.  In  the  Fully  Automated  condition, 
RoboLeader  provided  the  recommended  end  points  as  well  as  intermediate  waypoints  for  each  robot. 
The  participant  could  accept,  modify,  or  reject  the  plans.  In  each  scenario,  there  were  hostile  areas  (indicated 
by  red  squares  on  the  map)  that  the  robots  needed  to  avoid.  The  order  of  experimental  conditions  was 
counterbalanced  across  participants. 

The  study  was  a  within-subject  design  with  RoboLeader’ s  level  of  autonomy  as  the  independent  variable 
(with  four  levels:  Manual,  Semi-Autonomous  without  Visualization,  Semi-Autonomous  with  Visualization,  and 
Fully  Automated).  Dependent  measures  included  the  participants’  performance  of  encapsulating  the  primary 
target  (the  encapsulation  scores),  the  percentage  of  secondary  targets  (insurgents)  detected,  the  participants’  SA 
of  the  mission  environment  (percentage  of  SA  queries  answered  correctly),  and  the  participants’  perceived 
workload.  A  repeated-measure  analysis  of  variance  with  RoboLeader  as  the  within-subject  factor  was  used  to 
evaluate  the  operator  performance  differences  among  the  four  conditions. 
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Results  showed  that  RoboLeader  (Fully  Automated  condition)  was  more  effective  in  encapsulating  the 
moving  targets  than  were  the  human  operators  (when  they  were  either  without  assistance  from  RoboLeader  or 
when  they  were  partially  assisted  by  RoboLeader).  Participants  successfully  encapsulated  the  moving  targets 
only  63%  of  the  time  in  the  Manual  condition  but  89%  of  the  time  when  they  were  assisted  by  RoboLeader. 
Those  participants  who  played  video  games  frequently  demonstrated  significantly  better  encapsulation 
performance  than  did  infrequent  gamers;  they  also  had  better  SA  of  the  mission  environment.  Visualization 
had  little  effect  on  participants’  performance.  Finally,  participants  reported  significantly  higher  workload 
when  they  were  in  the  Manual  condition  than  when  they  were  assisted  by  RoboLeader. 

The  difficulty  levels  of  the  tasks  in  the  current  study  were  fairly  moderate.  Instead  of  comparing  the  4-robot 
and  8-robot  conditions,  the  study  could  have  investigated  the  effect  of  task  difficulty.  Different  outcomes 
could  have  been  observed  in  terms  of  the  effectiveness  and  usefulness  of  RoboLeader. 


16.9  LESSONS  LEARNED 

The  lessons  learned  were  many.  Specifically  for  agents  with  less  than  perfect  reliability  having  an  easily 
verifiable  display  space  mitigated  problems  with  false  alarms  but  not  misses  for  the  primary  task  of  target 
identification.  Most  interesting  was  the  superiority  of  experienced  gamers  for  overall  situation  awareness. 
Future  experiments  will  investigate  mitigating  factors  for  situation  awareness  well  as  target  identification. 


16.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  experiment  was  conducted  in  a  virtual  environment,  not  with  actual  robotic  vehicles. 


16.11  CONCLUSIONS 

We  concluded  that  future  battlefields  will  be  rife  with  manned  and  unmanned  vehicles  that  will  overwhelm  the 
Soldiers’  ability  to  conduct  their  assigned  missions  effectively  unless  technologies  are  developed  to  alleviate 
their  multi-tasking  requirements.  This  is  particularly  important  because  logistic  efficiency  will  require  Soldiers 
to  conduct  their  missions  in  a  many-to-one  configuration.  The  purpose  of  the  RoboLeader  simulations  was  to 
understand  how  to  develop  a  synergistic  relationship  between  human  supervisors  having  final  decision  authority 
and  intelligent  agents  who  supplies  algorithmic  solutions  for  many-to-one  control  problems.  The  initial 
experiment  established  the  feasibility  of  using  RoboLeader  to  control  up  to  eight  robots  during  a  reconnaissance 
mission.  The  second  experiment  focused  on  RoboLeader’ s  reliability  level  and  type  of  possible  errors. 
Surprisingly,  operators  were  able  to  intervene  more  successfully  with  False  Alarm  Prone  (FAP)  error  rates  than 
with  Miss  Prone  (MP)  error  rates  contrary  to  previous  findings  in  the  literature.  The  apparent  reason  for  this  was 
that  the  more  compact  interface  used  in  the  current  experiment  allowed  FAP  verification  to  be  accomplished 
more  efficiently.  In  contrast,  MP  errors  required  operators  to  constantly  scan  the  map  reducing  their  target 
detection  scores  on  the  video  displays.  The  final  experiment  showed  the  efficacy  of  RoboLeader  in  aiding  the 
operator  conduct  more  complex  missions  which  required  four  robots  to  entrap  a  moving  target. 

16.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

The  capabilities  of  RoboLeader  are  currently  being  expanded  to  deal  more  specifically  with  dynamic 
re-tasking  requirements  based  on  battlefield  developments  (e.g.,  individual  robots  need  to  be  re-tasked  to 
search  for  a  high-stake  target)  for  persistent  surveillance  in  urban  environments. 
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17.1  DATES 

The  US  Navy  is  investigating  the  usability  of  Remotely  Piloted  small  Unmanned  Surface  Vessels  (USVs)  to 
support  mine  warfare  missions.  The  demonstration  focuses  on  the  Human-Computer  Interface  (HCI)  for  the 
Multi-Robot  Operator  Control  Unit  (MOCU)  software  developed  at  the  Space  and  Naval  Warfare  Systems 
Center  Pacific  in  San  Diego  California  USA.  Human  performance  studies  are  being  conducted  to  investigate 
alternate  design  configurations  relative  to  optimum  human  performance  and  decision-making.  In  April  2009, 
the  first  technology  demonstration  illustrated  two  versions  of  the  HCI,  a  baseline  version  and  an  integrated 
map-video  version.  A  third  version  was  created  following  initial  end-user  review  of  the  map-video  version 
and  additional  testing  was  completed  in  March  2011  to  allow  comparison  of  results  in  a  dynamic  simulation 
across  the  versions.  The  goal  of  the  technology  is  to  safely  and  efficiently  control  simultaneous  operations  of 
two  USVs  and  to  identify  human  performance  shortcomings  that  may  be  mitigated  by  advanced  HCI 
concepts. 

HFM-170  Concept  Demonstration  1  was  conducted  April  22  2009.  Concept  2  usability  testing  was  performed 
August  5  -  12,  2010,  and  Concept  3  usability  testing  was  performed  February  28  -  March  10,  2011.  Further 
testing  is  planned  for  summer  2011. 


17.2  LOCATION 

Concept  Demonstration  1  was  conducted  April  22  2009  at  the  US  Naval  Submarine  Base  Point  Loma, 
San  Diego  California.  Concept  2/3  usability  testing  was  conducted  in  the  User  Laboratory  of  SPAWAR 
Systems  Center  -  Pacific,  San  Diego  CA. 


17.3  SCENARIO/TASKS 

The  USV  HCI  test  scenario  consisted  of  a  simple  navigation  route  tracking  which  simulated  both  ingress  and 
egress  from  a  “host  ship”,  e.g.,  the  US  Littoral  Combat  Ship  (LCS)  to  an  operational  area.  The  USV  scenario 
picked  up  after  a  simulated  post-launch  from  the  host  ship  at  a  control  point  where  the  on-deck  launch 
operator  would  hand-off  the  launched  USV  to  the  Organic  Off-board  Vehicle  Operator  (OOVO).  The  MOCU 
HCI  allowed  the  user  to  shift  between  automatic  waypoint-following  control  mode  to  manual  control  mode. 
Upon  completion  of  the  test  scenario  the  USV  would  begin  a  slow  search  pattern  for  mine  hunting. 
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The  remaining  mission  phases  were  deemed  to  duplicate  the  first  phase  for  human  performance  impact  and 
not  included  in  testing.  These  phases  included  mission  sensor  search  and  transit  to  a  recovery  dock  for 
retrieval. 

The  scenario  used  to  test  Concepts  2  and  3  in  2010  and  2011  followed  a  simulated  mission  scenario  with 
simulated  USVs.  Participants  in  both  sessions  were  USN  enlisted  personnel,  most  of  whom  had  previous 
experience  operating  USVs.  The  scenario  required  them  to  respond  to  a  series  of  pre-determined  conditions  and 
events  as  they  transited  two  USVs  from  the  host  ship  to  the  mission  operations  area.  The  scenario  was  designed 
to  elicit  performance  of  critical  tasks  derived  during  previous  task  analysis  interviews  with  subject  matter 
experts.  In  several  instances  scenario  events  were  purposely  scheduled  in  close  time  proximity  to  heighten 
mental  workload  and  assess  attention  management  capabilities  under  challenging  conditions.  Performance 
measures  were  developed  for  each  task,  which  usually  specified  a  window  of  time  for  completion.  The  scenario 
required  the  subjects  to  perform  the  tasks  listed  below: 

•  Take  control  of  USVs; 

•  Download  and  execute  pre-planned  routes; 

•  Set  emergency  maneuver  actions; 

•  Activate  radar  and  display  contacts; 

•  Switch  between  driving  modes  (manual  and  auto); 

•  Start/stop  engines,  including  set  to  idle; 

•  Drive  USV  in  manual  mode; 

•  Make  waypoint  reports  at  each  waypoint; 

•  Monitor  for  and  report  contacts  (radar  and  visual); 

•  Respond  to  stationary  contacts  in  path  (emergency  and  non-emergency); 

•  Respond  to  moving  contacts  in  path  (emergency); 

•  Respond  to  vessel  in  pursuit; 

•  Use  Point,  Tilt  and  Zoom  (PTZ)  camera  to  assess  contacts;  and 

•  Report/respond  to  system  status  alarms. 


17.4  TECHNOLOGIES  EXPLORED 

Demonstration  Concept  1  utilized  the  MOCU  HCI  with  a  live  robotic  USV.  Later  testing  used  the  HCI  with 
two  simulated  robotic  USVs.  In  all  cases  the  technology  focus  was  on  visualization  strategies  and  dynamic 
visual  and  audio  feedback  during  user  monitoring  and  control  of  missions.  In  addition  to  visualization 
methods,  use  of  hand-held  controller  technologies  was  investigated  in  Concept  3. 

The  integration  of  visualization  methods  within  the  HCI  challenges  the  end-users  visual  workload  and 
attention  management  skills  by  the  use  of  several  sets  of  cameras  with  various  visual  focus  domains.  These 
domains  include: 

1)  Pan-tilt-zoom  camera; 

2)  360  degree  camera;  and 
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3)  Rear- view  focus  camera  (rear- view  available  on  some  US  Vs). 

Other  sensory  inputs  include  Digital  Nautical  Charts  (DNC)  that  display  information  about  known  geographic 
features  including  algorithms  that  compare  objects  on  the  chart  to  the  depth  of  the  USV  keel  to  determine  if  a 
hazard  exists.  Color  coding  on  the  chart  display  indicates  nearby  geographic  hazards.  Radar  returns  are 
overlaid  with  the  digital  chart  information. 

The  HCI  concept  demonstration  was  conducted  using  a  PC-based  simulation  program  developed  by  the  Space 
and  Naval  Warfare  System  Center  Pacific  Unmanned  Systems  Group.  The  simulator  incorporated  video  graphics 
from  a  customized  commercial  software  nautical  gaming  simulator  integrated  with  the  MOCU-based  user 
interface  displays  and  controls.  In  Concepts  1  and  2  video  graphics  simulating  forward/aft/starboard  and  port 
camera  views  (as  well  as  PTZ)  for  each  of  the  two  US  Vs  were  displayed  on  the  upper  console  monitor  with 
USV#1  displayed  on  the  left  and  USV#2  displayed  on  the  right  (see  Figure  17-1).  The  lower  monitor  included  an 
integrated  Digital  Nautical  Chart  (DNC)  showing  landmasses,  radar  contacts  as  well  as  routes  and  waypoints  for 
both  USVs.  (See  Figure  17-2)  Operational  information  (speed,  heading,  location)  was  shown  for  each  USV. 
In  Concept  3,  video  graphics  were  displayed  in  an  integrated  “windshield”  style  display  (see  Figure  17-3)  on  the 
lower  console  and  the  DNC  was  displayed  on  the  upper  console  (see  Figure  17-4). 


Figure  17-1:  Upper  Display  for  Concept  2  Baseline  Version  of  MOCU  Multi-USV  Video  Information. 
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Figure  17-2:  Lower  Display  for  Concept  2  Baseline  Version  of  MOCU  Multi-USV  Chart  Information. 
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Figure  17-3:  Upper  Display  for  Concept  3  Version  of  MOCU  Multi-USV  Chart  Information. 
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Figure  17-4:  Lower  Display  for  Concept  3  Version  of  MOCU  Multi-USV  Video  Information. 


The  modularity  and  flexibility  of  the  MOCU  software  architecture  allows  for  relatively  quick  turnaround  in 
implementing  design  improvements,  therefore  the  software  is  well  suited  to  an  iterative  test  and  development 
effort. 


17.5  HUMAN  FACTORS  ISSUES  EXPLORED 

Human  Factors  performance  issues  with  two  simultaneous  semi-autonomous  USVs  include  the  following: 

1)  Attention  Management  and  Attention  Allocation  -  Autonomous  systems  such  as  USVs  that  may 
be  in  fully  auto  or  fully  manual  control  modes  require  the  user  to  know  where  and  how  long  to  focus 
on  information  pertaining  to  each  USV.  Also,  the  user  must  shift  attention  between  USVs.  The  user’s 
strategy  must  be  aligned  with  the  environment  (e.g.,  traffic  congestion)  and  speed  of  the  USV  and 
mission  tempo  (pace  of  mission  events).  Attention  management  and  human  vigilance  is  subject  to 
errors  in  allocation  and  fatigue.  Initial  tests  of  the  baseline  1  model  in  2008  indicated  the  probably  of 
error  was  high  and  that  visual  feedback  in  terms  of  type  of  information  coding  was  not  adequate. 
Also,  the  point-and-click  type  of  control  implementation  required  full  visual  attention.  Further  testing 
in  2010  indicated  significant  performance  decrement  issues  if  baseline  visual  cues  were  used. 
The  Concept  3  version  shown  in  Figure  17-3  and  Figure  17-4  included  the  reconfiguration  of  displays 
and  use  of  additional  visual  cues. 

2)  Mental  Model  of  Robot  and  Mission  State  -  The  user  must  maintain  an  awareness  of  USV  mission 
status  and  USV  equipment  status.  Situation  Awareness  includes  an  accurate  mental  model  of  mission 
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objectives  (reaching  waypoints,  deploying  sensors)  and  safety  (approaching  danger  from  fixed  or 
moving  objects).  The  test  scenarios  for  Concepts  2  and  3  included  verbal  reports  for  mission 
waypoints  to  superior  officers.  The  verbal  reporting  activity  adds  to  overall  user  workload.  Position 
awareness  and  orientation  requires  understanding  of  each  USV  relative  to  the  ship  platform,  and  with 
two  US  Vs  potentially  three  different  course  and  speeds  simultaneously. 

3)  Performing  Emergency  Maneuvers  -  The  user  may  need  to  respond  to  an  unexpected  safety  issue 
or  threat  requiring  a  shift  from  automated  to  manual  control  and  a  corresponding  course  and  speed 
change  to  avoid  collision.  If  the  user  cannot  quickly  orient  and  respond  to  an  emergency  event  the 
mission  and  USV  could  be  at  significant  risk  of  collision  and  mission  failure. 

Correlation  of  real-world  stimulus  associated  with  multiple  camera  views  has  been  a  significant  design 
challenge.  The  camera  views  distort  the  perception  of  approaching  and  crossing  objects  (e.g.,  other  vessels). 
A  successful  HCI  design  requires  an  integration  of  information  that  minimizes  visual  scanning  and  shift  from 
one  display  to  another.  The  design  problem  for  afloat  US  Vs  differs  from  both  unmanned  air  and  surface 
(ground  robots)  in  the  dimensions,  approach  and  characteristics  of  obstacles  and  ability  to  detect  and  avoid 
obstacles.  The  water  operational  space  is  not  controlled  as  air  space  is  and  the  water  surface  is  constantly 
moving  with  waves  and  floating  objects,  including  submersed  objects. 

To  mitigate  human  performance  risk  of  errors  and  improve  performance  efficiency,  several  enhanced  design 
attributes  were  implemented  and  tested.  These  attributes  include: 

1)  Attention  cues  to  aid  in  shifting  of  attention  between  US  Vs. 

2)  Orientation  of  map  display  and  camera  views  to  provide  synchronized  visual  feedback  to  aid  in 
maintaining  an  ongoing  mental  model  of  USV  position. 

3)  Improved  visual  feedback  as  mission  waypoints  approach  and  are  passed. 

4)  Integration  of  a  hand-held  “game”  controller  to  replace  point-and-click  methods  to  reduce  visual 
workload  associated  with  manual  control.  This  allows  the  users  visual  resources  to  maintain  a  camera 
view  focus  while  a  maneuver  is  made. 

5)  Overlay  and  integration  of  key  status  information  with  ongoing  dynamic  information  from  cameras 
and  maps  to  reduce  visual  search  and  scanning. 

6)  USV-specific  color  coding  of  video  display  window  borders  and  vessel  status  information  to  reduce 
confusion  between  US  Vs. 

17.6  UNMANNED  SYSTEMS  USED 

The  live  demonstration  used  a  laboratory  model  USV  that  was  comprised  of  a  commercial  craft  modified  for 
remote  control  radio  with  sensors  mounted  onboard. 
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Figure  17-5:  Unmanned  Surface  Test  Vehicle. 


The  following  components  are  part  of  the  demonstration: 

•  Unmanned  Surface  Test  Vehicle  is  a  lightweight  Length:  20’  6”  /  6.25  m,  weight:  3,250  lbs  /  1,474  kg 
consumer  (non-ruggedized)  craft  containing  sensor  packages. 

•  Sensors  include:  Digitized  marine  radar,  Video  (stabilized  or  non-stabilized),  Stereovision  (3D  range 
data),  Monocular  vision  for  obstacle  detection. 

•  Automatic  Identification  System  (AIS)  (receive-only  currently,  Uncooled  thermal  imager  (in  future 
possible  laser  range  scanner). 

•  Multi-robot  Operator  Control  Unit  (MOCU)  baseline  version  software  and  HCI  package  enabling  user 
monitoring  and  control  of  one  or  more  USVs. 

•  Radio  transmitter  and  receiver  for  video  and  communications  to/from  vehicle. 

•  Obstacle  detection  and  avoidance  software  and  methods  were  disabled. 

For  simulated  tests  in  the  laboratory,  the  MOCU  simulation  used  the  larger  11 -meter  “fleet”  class  USVs 
designed  for  operational  missions.  The  simulation  also  replicated  the  types  of  cameras  available  on  the 
operational  model.  These  USVs  weigh  approximately  7700  kilograms.  The  USVs  are  designed  to  be  remotely 
operated  from  the  LCS  host  ship.  Although  each  USV  will  be  equipped  with  radar,  current  plans  do  not 
include  onboard  obstacle  avoidance  capability. 


17.7  SUMMARY  OF  ANY  NATO  COMMUNICATIONS/COLLABORATIONS/ 
INTERACTIONS 

1)  USN  received  design  guidelines  from  Canada  Defence  R&D  Canada  -  Toronto  on  Intelligent  Adaptive 
Systems. 
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2)  Received  guidelines  on  visual  display  symbology  from  US  Army  HFM-170  member. 

3)  Posted  guidelines  for  command  and  control  mission  flow  visualization  to  all  members. 

4)  USN,  Canada,  Netherlands  -  discussed  generalization  of  results  to  Explosive  Ordnance  Disposal  robot 
applications. 

5)  USN,  USAF  -  Discuss  speech  and  voice  technology  applications  for  robot  control. 

6)  USN,  US  Army,  Discussion  of  playbook  and  work  process  visualization  for  mission  supervision. 


Planning/Design 

Execution 

Analysis 

Communication 

1)  USN  and  Canada 

2)  USN,  Canada,  NE 

Coordination 

2)  USN  and  US  Army 

Collaboration 

2)  USN  and  all 

Potential  USN  and  US  Army 
future  collaboration  in 
upcoming  cargo  air  robot 
project 

17.8  SUMMARY  OF  TECHNICAL  DEMONSTRATION  RESULTS 

The  operational  demonstration  was  successfully  completed  in  April  2009  [1],[2]  using  the  integrated  map  and 
video  version  of  MOCU.  The  demonstration  had  several  caveats  regarding  validity  of  results.  First,  the  users 
were  not  end-users  (Navy  operators)  but  instead  were  project  engineers.  Second,  the  mission  scenario  and 
course  was  limited  to  a  small  range  and  area  due  to  safety  precautions,  with  a  live  human  operator  available 
on  the  USV  to  take  over  control  in  case  of  emergency.  Overall  the  demonstration  showed  the  validity  of  the 
initial  HCI  design  concepts  and  demonstrated  a  test  capability  to  conduct  further  testing  and  analysis. 

Subsequent  testing  sessions  in  2010  [3], [4]  and  2011  [5]  evaluated  Concept  2  (baseline)  and  Concept  3  HCI 
designs  in  support  of  multiple  robot  operations.  Overall,  the  testing  showed  that  Navy  operators  had 
difficulties  in  attending  to  two  US  Vs  simultaneously,  however  performance  was  significantly  improved  for 
Concept  3  with  the  addition  of  the  enhanced  design  attributes  described  above.  User  performance  across  three 
task  areas  is  discussed  below: 

•  Responding  to  contacts  in  emergency  situations  -  Each  scenario  included  four  events  requiring  the 
subjects  to  observe  and  maneuver  around  one  or  more  vessels  stationed  or  moving  in  the  direct  path 
of  one  of  the  USVs.  In  Concept  2  testing,  all  subjects  failed  to  avoid  collision  in  at  least  one  instance 
and  most  were  involved  in  multiple  collisions,  resulting  in  an  overall  collision  rate  of  67%.  Subjects 
typically  noticed  the  contacts  too  late  to  take  effective  evasive  action  -  even  though  all  contacts  were 
visible  in  at  least  one  video  window  for  at  least  30  seconds  prior  to  impact.  In  three  instances,  subjects 
failed  to  notice  the  contact  at  all  and  took  no  evasive  action  whatsoever  as  they  were  monitoring  other 
video  windows  or  performing  other  tasks.  In  Concept  3  testing,  subjects  not  only  showed  an  improved 
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capability  to  detect  contacts  but  also  demonstrated  improved  capability  to  successfully  execute 
avoidance  actions,  resulting  in  a  decline  in  the  overall  collision  rate  to  27%.  This  rate  is  still  significant 
for  operational  conditions.  Thus,  an  upgraded  Concept  3.1  will  be  generated  and  tested. 

•  Responding  to  system  status  alarms  -  During  the  scenario  a  system  status  alert  was  turned  red  and 
flashed  indicating  a  high  engine  temperature  alarm.  Operators  responded  by  making  a  report  to  the 
mission  supervisor.  In  Concept  2  testing,  reports  were  made  within  the  designated  response  window 
only  33%  of  the  time.  In  Concept  3  testing  however,  the  response  rate  improved  to  63%  with 
improvements  also  noted  in  the  ability  of  subjects  to  take  appropriate  action  (i.e.,  shut  down  engines) 
in  a  timely  manner. 

•  Making  verbal  waypoint  reports  to  command  -  Operators  were  directed  to  make  reports  to  the 
mission  supervisor  as  each  waypoint  was  reached  along  the  pre-planned  routes.  In  Concept  2  testing, 
timely  reports  were  made  90%  of  the  time  for  the  first  and  last  (4th)  waypoints  on  the  routes  (typically 
reached  during  times  of  low  scenario  activity  when  subjects  had  few  distractions),  and  77%  of  the 
time  for  waypoints  2  and  3  (which  occurred  during  periods  of  high  scenario  activity  and  heightened 
mental  workload).  In  Concept  3  Testing,  subjects  completed  waypoint  reports  successfully  97%  of  the 
time  for  waypoints  1  and  4,  and  84%  of  the  time  for  waypoints  2  and  3. 

In  addition  to  the  improved  performance  noted  on  objective  measures  for  the  Concept  3  HCI  design, 
subjective  measures  collected  during  an  exit  survey  also  showed  a  strong  user  preference  for  the  Concept  3 
controls  and  display  configuration. 


17.9  LESSONS  LEARNED 

The  initial  findings  of  this  study  demonstrated  that  the  baseline  Concept  2  interface  would  not  safely  support 
simultaneous  operation  of  multiple  US  Vs  and  identified  a  number  of  specific  opportunities  for  improving  the 
overall  HCI  that  were  incorporated  into  a  Concept  3  design.  Although  subsequent  testing  of  the  enhanced 
design  showed  dramatic  improvement  across  all  performance  measures,  operator  errors  were  still  observed  at 
an  unacceptable  level  and  additional  opportunities  for  design  improvements  were  noted,  including: 

•  Increased  collision  avoidance  aiding  tied  to  attention  alerting  cues  (the  need  for  advanced  obstacle 
cues  may  require  placement  of  additional  sensors  onboard  USV  platforms). 

•  Prominent  urgent  alarm  messaging,  to  include  the  addition  of  audio  alerts. 

•  Improved  color  coding  to  depict  route  graphics. 

•  Refinement  of  hand  held  controls  to  reduce  joystick  sensitivity  and  prevent  inadvertent  shifts  in 
driving  mode. 

•  Enhanced  indication  and  control  of  PTZ  camera  magnification  levels. 

•  Additional  engine  status  indication  and  independent  start/stop  controls. 

17.10  STUDY  CONSTRAINTS/LIMITATIONS 

The  primary  limitation  of  this  usability  testing  was  the  fidelity  of  the  simulator  and  the  realism  of  the  mission 
scenario.  Several  aspects  of  the  simulator  differ  from  the  actual  system  including: 

•  The  substitution  of  digital  animation  for  live  video; 
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•  Non-functionality  of  many  secondary  screens  that  operators  would  normally  have  access  to; 

•  The  actual  shipboard  hardware  with  alternate  video  monitors;  and 

•  Substitution  of  a  mouse  for  trackball  control. 

Although  every  attempt  was  made  to  build  a  realistic  mission  scenario,  it  must  be  recognized  that  the  initiating 
events  and  responding  actions  represented  in  the  scenario  would  in  actuality  unfold  over  several  hours  as 
opposed  to  the  30  minutes  it  took  to  simulate  the  mission.  When  questioned  about  the  realism  of  the 
simulation  most  subjects  (including  the  most  experienced  USV  operator  who  was  involved  in  developing  the 
Operational  Procedures  for  the  real  system)  indicated  that  it  was  “pretty  close”  or  “not  far  off’  and  that 
fidelity  level  would  be  sufficient  to  serve  as  a  “useful  training  aid”. 


17.11  CONCLUSIONS 

The  results  of  the  usability  studies  confirmed  the  existence  of  many  human  factors  concerns  that  had  been 
identified  through  previous  heuristic  reviews  and  HCI  design  walkthroughs  of  interface  displays.  Based  on 
the  results  of  the  testing  in  which  even  experienced  operators  demonstrated  degraded  performance, 
the  researchers  concluded  that  the  baseline  Concept  2  design  interface  would  not  safely  support  simultaneous 
operation  of  two  USVs.  Design  Concept  3  shows  great  promise  but  is  not  yet  at  a  level  that  would  support  safe 
and  reliable  operation  of  multiple  USVs  simultaneously.  Concept  3.1  will  be  generated  and  tested  in  201 1. 


17.12  FUTURE  RESEARCH  NEEDS  AND  PLANS  IN  THIS  AREA 

Further  design  alternatives  will  be  explored  in  developing  an  improved  interface  that  will  be  tested  and 
compared  to  previous  versions.  The  goal  of  the  studies  will  be  to  measure  performance  of  the  current  USV 
sensor  package,  with  the  HCI  improvements.  Another  configuration  will  include  obstacle  avoidance  aids  that 
are  technically  feasible.  These  aids  will  be  simulated  and  tested  for  comparison  with  the  lower  cost,  lower 
fidelity  sensor  package.  A  design  trade-off  between  USV  cost,  risk  and  user  performance  can  then  be 
accomplished. 
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This  underlying  report  describes  how  HFM-170  successfully  developed  and  demonstrated  pertinent  supervisory 
control  human-system  interface  design  practices  and  operator  interface  concepts  for  Uninhabited  Vehicles  (UVs) 
network-centric  operations.  In  a  series  of  14  specific  Technology  Demonstrations  (TDs)  it  was  shown  that  the 
operator’s  role  is  becoming  more  supervisory  of  nature  since  future  UVs  will  be  increasingly  automated 
(e.g.,  autonomous  capabilities,  multiple  systems,  systems  of  systems),  on  the  other  hand  it  was  demonstrated  that 
new  sensor  and  control  technologies  enable  operators  to  be  closer  in  the  loop  in  a  telepresence  situation. 
The  applications  addressed  varied  in  degree  of  autonomy  from  manual  robotic  control  to  highly  autonomous, 
swarming  UVs.  A  variety  of  critical  issues  were  addressed  including  multi-vehicle  control,  manned-unmanned 
teaming,  human-automation  interaction,  telepresence  interfaces,  delegation  interfaces,  vehicle  hand-offs, 
operator  workload  adaptive  systems,  variable  levels  of  autonomy,  authority  sharing,  situation  awareness  aids, 
cognitive  workload  assessment,  swarming  interfaces,  and  dynamic  mission  management.  HFM-170  also 
concentrated  on  the  identification  and  demonstration  of  successful  supervisory  control  methodologies  and 
interface  design  practices  for  enabling  single  operator  control  of  multiple  UVs. 

Based  on  these  TDs  the  following  conclusions  can  be  drawn: 

•  Live-demonstration  of  UV  supervisory  control  methods  and  enabling  technology  as  experienced 
within  NATO  HFM-170  is  a  complex  undertaking  which  requires  serious  preparation.  These  TDs 
were  well  received  by  the  entire  Task  Group.  [ALL] 

•  These  demonstrations,  along  with  periodic  meetings,  provided  a  valuable  forum  to  exchange  technical 
information  and  discuss  possible  future  collaborations  in  supervisory  control  research  and 
development.  [ALL] 

•  The  developed  7  Dimension  Framework  model  held  the  most  promise  for  satisfying  the  ends  of  the 
Task  Group.  This  model  was  largely  descriptive,  but  it  captured  several  dimensions  relevant  to  the 
many  alternate  supervisory  control  systems,  relationships  and  usages  we  were  examining.  While  the 
specific  dimensions  examined  need  to  be  refined,  and  the  scales  for  characterizing  them  might  also  be 
improved  upon,  this  multi-dimensional  description  of  alternate  systems  seemed  to  provide  the  right 
level  and  type  of  information  for  conveying  how  a  set  of  supervisory  control  systems  are  similar  and 
different  from  each  other.  [ALL] 

•  Hand-off  demonstrations  between  two  UV  supervisory  control  crews,  as  well  as  between  an  external 
pilot  (flying  manual  control)  and  a  supervisory  control  station  was  successfully  verified.  [CAN-1] 

•  Self-organization  and  protection  capabilities  of  multiple  autonomous  ground  vehicles  through  Artificial 
Impedance  Control  for  local  autonomy  including  collision  avoidance  and  trajectory  generation  showed 
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excellent  results.  Expanding  this  with  formation  control  and  flocking  control  through  computer 
simulation  showed  promising  results.  [CAN-2] 

•  A  technology  demonstration  called  OmniSense  showed  the  efficacy  of  a  multi-modal  display 
(i.e.,  the  presentation  of  visual,  auditory,  and  tactile  information)  for  enhancing  supervisory  control  of 
an  automated  UAV.  The  benefit  of  OmniSense  is  anticipated  to  be  particularly  evident  in  an  increase 
in  the  detection  of  critical  events,  a  reduction  of  response  times  to  critical  events,  and  increased 
situation  awareness.  This  suggests  that  the  OmniSense  solution  will  be  more  effective  than  a  visually- 
only  GCS  interface.  [CAN-3] 

•  Swarm  intelligence  seems  to  be  a  promising  approach  for  multiple  UVs  control  in  terms  of 
algorithmic  performance  and  robustness,  as  long  as  human  factors  and  especially  man-machine 
communication  and  interaction  are  properly  adapted.  [FRA-1] 

•  A  program  demonstrating  new  means  of  cooperation  and  interaction  between  Humans  and  Automates 
(“Authority  Sharing”)  showed  that  it  is  possible  to  optimize  the  workload  of  existing  UAV  systems 
by  allocating  dynamically  the  operators’  functions,  allowing  thus  the  integration  of  multiple  UAVs 
and  payloads  without  necessarily  increasing  the  number  of  operators  required  to  manage. 
The  operators  appreciated  the  human-machine  interface,  in  particular  the  “draggable  vector  tool”. 
Regarding  the  “authority  sharing”  engine,  the  overall  performance  does  not  change  with  or  without 
the  activation  of  the  engine,  but  the  test  panel  was  too  small  to  statistically  confirm  this  data.  [FRA-2] 

•  The  demonstration  of  a  generic  approach  in  the  development  of  a  knowledge-based  assistant  system 
adapted  to  the  domain  of  manned-unmanned-teaming  focused  on  guidance  of  multiple  UAVs  from 
the  commander’s  workplace  in  a  helicopter  cockpit  aided  by  an  assistant  system.  This  approach  was 
evaluated  through  experiments  in  the  helicopter  simulator.  The  introduction  of  the  assistant  system 
improved  human  factors  related  variables  like  situation  awareness  and  workload,  improved 
performance  and  safety,  and  was  well  accepted.  It  is  also  concludes  that  the  next  steps  to  further 
improve  the  assistant  system  performance  and  acceptance  should  be  to  refine  the  knowledge  models 
for  operator  overtaxing  estimation,  current  task  recognition  and  cost  prediction,  and  to  refine  the 
action  and  decision  support  for  tasks.  Finally,  the  cooperation  and  variable  task  assignment  between 
commander  and  pilot  flying  have  to  be  further  investigated  and  regarded  within  the  concept.  [GER-1] 

•  An  experimental  research  collaboration  between  the  US  Army  Research  Lab  and  NL  TNO  where 
robots  were  used  for  reconnaissance  of  a  remote  area  revealed: 

1)  No  difference  between  the  Mono-Headtracking  condition  and  the  Mono- Joystick  condition  in 
accuracy  of  target  identification.  However,  more  time  was  required  for  target  identification  when 
using  joystick  control. 

2)  The  Telepresence  condition  (that  included  3D  audio)  increased  the  percentage  of  correctly 
identified  targets  by  approximately  23%  compared  to  the  Mono-Headtracking  condition  (using  a 
directional  microphone).  In  addition,  target  identification  took  approximately  35%  longer  without 
having  the  3D  audio  functionality  available. 

3)  The  Telepresence  human-robot  interface  decreased  identification/localization  times  for  audio 
stimuli  by  approximately  42%  as  compared  to  currently  commonly  used  interfaces.  In  addition, 
target  identification  performance  increased  by  about  26%  when  using  the  Telepresence  human- 
robot  interface.  [NL-1] 

•  A  test  of  a  new  framework  for  managing  UAV  task  and  workload  allocation  between  various 
operators  in  a  mission  scenario  revealed  improvements  of  optimal  in-the-loop  inclusion  of  operators 
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for  the  successful  application  of  multi-UAV  systems.  Further  testing,  with  simulated  scenarios, 
will  provide  new  insights  over  the  real  capabilities  of  the  proposed  framework.  [PT-1] 

•  A  demonstration/test  where  an  operator  had  to  mainly  manually  navigate  one,  two,  or  three  partly 
autonomous  UGVs  to  pre-designated  inspection  points  in  a  simulated  urban  environment  showed  that 
the  limited  autonomous  function  was  insufficient  to  significantly  improve  the  operator  to  vehicle 
ratio.  The  operators  were  saturated  even  when  only  controlling  two  UGVs  in  a  basic  navigation  task. 
More  advanced  autonomous  functions  or  control  station  interfaces  that  reduce  the  attention  demands 
are  therefore  necessary  to  improve  the  operator  to  vehicle  ratio.  [SWE-1] 

•  A  demonstration/test  revealed  that  partly  autonomous  UGV  functions  can  improve  the  operator  to 
vehicle  ratio  if  there  are  weak  dependencies  between  the  task  that  an  operator  performs  and  the  task 
that  the  partly  autonomous  function  performs.  However,  these  task  dependencies  also  depend  on  the 
operators’  task  experience,  as  well  as  the  formal  task  properties.  The  operators’  control  strategy 
showed  that  they  were  rather  naive  towards  the  complexities  of  tactical  reconnaissance  tasks.  More 
experienced  operators  may  therefore  find  different  task  dependencies.  [SWE-2] 

•  Demonstration  and  test  of  the  Dynamic  Airborne  Mission  Management  (DAMM)  program  developed 
a  set  of  principles,  interfaces  and  interactions  for  the  delivery  of  effects,  enabled  by  advanced  digital 
networking  and  mission  enabling  technologies,  providing  a  distributed,  collaborative  and  adaptive 
mission  capability  for  stability  and  dominance  in  a  dynamic  environment.  DAMM  synthetic 
environment  and  flight  trials  provide  evidence  with  high  levels  of  proof  for  real  benefits  of  a  full  suite 
of  collaborative  decision  support  tools  across  multiple  tiers  of  the  networked  dynamic  Command  and 
Control  (C2)  architecture.  This  work  has  shown  that  as  data  rate,  confidence  and  context  increase/ 
improve,  nodal  (micro)  and  system  (macro)  C2  decision  loop  activity  transposes  from  slow  serial  to 
concurrent-NRT  speeds.  Consequently,  kill-chain  timeline,  fratricide  and  collateral  incidents  should 
reduce.  [UK-1] 

•  The  Multi-UAV  Supervisory  Control  Interface  Technology  (MUSCIT)  program  demonstrated  several 
advances  in  UAV  control  station  interface  technology  that  enables  effective  single-operator,  multi- 
UAV  performance  for  surveillance  and  re-routing  tasks.  Interface  enhancements  included  a  novel 
tactical  situation  (map)  display,  speech  recognition,  synthetic  overlays,  mission  and  sensor 
automation,  integrated  information  displays  tailored  for  supervisory  control,  and  support  tools  for 
multi-sensor  management  tasks.  The  control  station  technology  was  demonstrated  in  a  four- vehicle 
configuration  made  up  of  two  actual  UAVs  in  flight  along  with  two  simulated  UAVs,  all  controlled 
by  a  single  operator.  The  operator  interface  was  iteratively  developed  using  a  spiral  approach; 
combining  simulation  and  flight  testing  to  characterize  operator  and  mission  performance,  empirically 
derive  and  refine  technical  requirements,  and  refine  operator  interface  technology.  Empirical  results 
from  simulation  and  flight  evaluations  reveal  specific  costs  to  operator  performance,  situation 
awareness,  and  workload  as  a  result  of  increasing  the  number  of  UAVs  a  single  operator  is  required  to 
manage.  [US-1] 

•  Delegation  Control  of  multiple  heterogeneous  UVs  by  a  single  operator  was  successfully 
demonstrated.  Navigation  and  payload  control  of  four  unmanned  systems  was  monitored  by  a  single 
operator  in  a  collaborative  urban  mission  scenario.  Delegation  Control  employment  strategy  and 
interface  design  supported  the  build,  initiation,  modification  and  monitoring  of  simultaneous  plays  in 
progress.  Use  of  voice  recognition  was  considered  an  advantage  to  the  operator  during  time  critical 
mission  phases.  The  operator’s  ability  to  bypass  menus  in  favor  of  voice  recognition  control, 
significantly  decreased  reaction  time  to  external  mission  events.  Dynamic  route  re-  planning  was 
effectively  accomplished  while  plays  were  in  progress.  In  addition,  play  status  was  efficiently 
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depicted  and  real-time  updates  were  accomplished  when  play  modification  and  play  terminations 
occurred.  Automation  transparency  was  increased  through  messages  that  described  impacts  of  conflicting 
plays.  Lastly,  the  Play  Status  window  was  considered  a  significant  contribution  to  operator  situation 
awareness  for  rapid  awareness  of  asset  allocation  and  play  scheduling.  [US-2] 

•  The  results  of  two  experiments  using  RoboLeader,  using  simulations  to  understand  how  to  develop  a 
synergistic  relationship  between  human  supervisors  (having  final  decision  authority)  and  intelligent 
agents  (who  supplies  algorithmic  solutions  for  many-to-one  control  problems)  showed  that  operators 
were  able  to  intervene  more  successfully  with  False  Alarm  Prone  (FAP)  error  rates  than  with  miss 
Prone  (MP)  error  rates,  contrary  to  previous  findings  in  the  literature.  The  apparent  reason  for  this 
was  that  the  more  compact  interface  used  in  the  current  experiment  allowed  FAP  verification  to  be 
accomplished  more  efficiently.  In  contrast,  MP  errors  required  operators  to  constantly  scan  the  map 
reducing  their  target  detection  scores  on  the  video  displays.  The  final  experiment  showed  the  efficacy 
of  RoboLeader  in  aiding  the  operator  to  conduct  more  complex  missions  which  required  four  robots 
to  entrap  a  moving  target.  [US-3] 

•  The  results  of  usability  studies  to  investigate  alternate  design  configurations  relative  to  optimum 
human  performance  and  decision-making  in  USV  supervisory  control  confirmed  that  the  baseline 
concept  design  interface  showing  video  graphics  simulating  forward/aft/starboard  and  port  camera 
views  for  each  USV  would  not  safely  support  simultaneous  operation  of  two  US  Vs  by  a  single 
operator.  A  design  concept  displaying  an  integrated  “windshield”  style  display  showed  great  promise 
but  is  not  yet  at  a  level  that  would  support  safe  and  reliable  operation  of  multiple  US  Vs 
simultaneously.  The  latter  concept  will  be  further  tested.  [US-4]. 

A11  TDs  were  very  successful  and  well  received  by  the  Task  Group  members.  It  is  therefore  recommended  to 
disseminate  results  and  lessons  learned  associated  with  the  technical  demonstrations  of  HFM  Technical  Task 
Group  HFM- 170,  Supervisory  Control  of  Multiple  Uninhabited  Systems  -  Methodologies  and  Enabling 
Human-Robot  Interface  Technologies.  The  aim  is  to  bring  together  representatives  of  the  research  and 
operational  communities  at  invitation,  to  present  technical  demonstration  results,  and  to  review  progress  in 
this  important  area. 
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A.l  DYNAMIC  AIRBORNE  MISSION  MANAGEMENT 

Fundamentally,  DAMM  is  primarily  concerned  with  adaptation  of  mission  command,  mission  flow  and  effects 
delivery  to  changes  in  the  mission  context.  In  UK  MOD  operations,  the  mission  context  for  tactical  missions  is 
customarily  briefed  in  terms  of  the  “4  Ts”  -  Tasks,  Targets,  Threats,  Tactics  -  and  the  observed  impact  on 
timeliness  for  a  co-ordinated  and  precision  engagement  mission.  Thus,  the  4T’s  provide  an  operationally  relevant 
representation  and  high  level  decomposition  of  the  key  elements  of  the  mission  context.  A  simple  representation 
of  the  functional  flow  model  for  DAMM  in  relation  to  the  4Ts  is  shown  in  Figure  A-l  below. 


Timeliness  ATO/ACO 


Figure  A-1:  Functional  Model  of  DAMM. 


The  functional  flow  model  of  DAMM  illustrated  in  Figure  A-l  provides  a  useful  framework  for  planning  of 
DAMM  test  and  evaluation  studies.  This  framework  has  utility  for  defining  test  variables  and  metrics, 
with  potential  discriminative  power  for  diagnostic  and  prognostic  analysis.  All  4T’s,  coupled  with  command 
intent  and  the  timeliness  of  effects,  should  be  considered  as  essential  mission  variables  and  sources  of  metrics 
for  comprehensive  studies  of  DAMM  advanced  digital  networking  and  mission  enabling  technologies. 
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A.2  MISSION  CONTEXT 

Measurement  and  control  of  the  complexity  of  the  mission  context  provides  a  basis  for  standardisation, 
comparison  and  balance  in  test  design.  The  “4  T’s”  -  Tasks,  Targets,  Threats,  Tactics  -  with  timings  of  effects, 
provide  an  operationally  relevant  framework  for  the  description,  decomposition,  and  measurement  of  the  mission 
context.  The  frequency  of  individual  tasks,  targets,  threats,  tactics  (and  affected  timings)  can  be  controlled, 
observed  and  measured  directly.  Additionally,  given  the  familiarity  of  operators  with  the  4T’s  framework  for 
briefing  missions,  it  seems  sensible  and  potentially  useful  to  try  to  elicit  from  participant  operators,  or  from 
observer  subject-matter  experts,  estimates  of  the  demands  on  operators  workload  arising  from  changes  in 
mission  T’s.  Accordingly,  in  the  both  1st  and  2nd  US-UK  Strike  Warrior  SE  Trials,  rating  scale  estimates  (using 
7-point  Likert  scales)  of  Change  Management  Demand  for  Tasks,  Targets,  Threats,  and  Tactics  (Times  1st  Trial 
only)  were  obtained  from  the  participants  for  individual  trial  runs.  The  mission  T’s  change  demand  ratings  data 
showed  evidence  of  systematic  and  sensible  trends  (Tasks>Targets>Threats>Tactics>Times)  and  some 
statistically  significant  beneficial  effects  of  advanced  system  architectures  (Baseline>Threshold>Objective). 
It  was  noted  that  Target  and  Threat  demands  arose  directly  from  the  external  environment  and  mission  scenario. 
In  contrast,  Tasks,  Tactics  and  Times  were  mitigation  responses  mediated  by  the  system  architectures  and 
mission  management. 


A.3  SCALE 

In  the  operational  environment,  DAMM  involves  complex  interactions  and  interfaces  between  air  packages, 
C2  elements  and  air-land  co-ordination.  In  planning  realistic  technology  demonstrations  and  operational 
testing,  the  scale  of  the  tested  operations  and  architectures,  and  the  degree  of  uncertainty  or  volatility  in  test 
missions,  are  major  determinants  of  the  validity,  reliability  and  generalisability  of  test  findings.  Scale  is  a 
major  study  cost  driver.  More  affordable  small-scale,  sub-system  studies  provide  simpler  effects  and  easier 
measurement,  but  risk  low  generalisability  of  findings.  More  costly  large  scale,  system-of-system  realistic 
demonstrations  can  be  convincing  and  impressive,  but  the  more  complex  effects  arising  can  be  difficult  to 
quantify  and  verify,  in  particular  with  regard  to  repeatability  and  reliability.  A  mixed  approach  is  probably 
preferable,  using  progressive  development  and  testing  of  prototypes,  for  better  managing  the  risks  and  costs  of 
scale.  This  can  be  provided  by  a  series  of  development  and  test  phases,  with  increasing  complexity, 
and  prioritisation  of  core  capabilities,  critical  interactions  and  essential  interfaces.  A  progressive  approach  can 
be  facilitated  by  exploiting  any  inherent  scalability  in  the  technical  system  and  testing  scenarios.  The  DAMM 
architecture  afforded  progressive  building  and  extension  of  the  horizontal  (effectors  packages)  and  vertical 
(tactical/operational  command)  C2-MM  system  components.  The  DAMM  scenario  afforded  incremental 
development  of  mission  complexity  by  the  addition  of  tasks,  targets,  and  threats. 


A.4  VOLATILITY  AND  UNCERTAINTY 

DAMM  seeks  to  enable  adaptation  and  stability  in  a  dynamic  environment.  The  scenario  and  missions  were 
designed  to  allow  White  Force  to  vary  volatility  and  create  uncertainty  through  injection  of  unexpected  and 
disruptive  information  and  events  via  tasks,  targets  and  threats.  Variability  in  the  volatility  and  uncertainty  of 
the  missions  is  necessary  to  stress  and  test  human  component  capabilities: 

•  To  exercise  and  challenge  the  operator’s  use  of  DAMM  tools,  and  application  of  skills,  rules  knowledge 
underpinning  Mission  Essential  Competencies  (MEC); 

•  To  mitigate  operator  learning; 


A  -  2 


RTO-TR-HFM-1 70 


^PNATO 
WP  I  OTAN 


ANNEX  A  -  DYNAMIC  AIRBORNE 
MISSION  MANAGEMENT:  LESSONS  LEARNT 


•  To  provide  military  operational  variety  and  realism;  and 

•  To  enable  participant  engagement  and  immersion. 

Control  of  the  scale  of  volatility  and  uncertainty  in  the  test  scenario  missions  provides  a  further  basis  for 
standardisation,  comparison  and  balance  in  test  design.  The  frequency  of  injections  of  changes  affecting  tasks, 
targets,  threats,  and  tactics  afforded  by  the  vignettes  can  be  observed  and  measured  directly  to  provide  direct 
measurement  of  the  mission  volatility.  Estimates  of  Change  Management  Demands  associated  with  the  4Ts 
provide  indirect  measurements  of  the  resulting  uncertainty.  However,  these  are  confounded  with  the  mitigating 
effects  of  the  DAMM  system  architectures.  The  scenarios  were  designed  with  a  set  of  vignettes  (typically  3+)  to 
provide  variety  and  challenge,  with  progressive  complexity  and  volatility.  In  practice,  the  degree  of  volatility  and 
uncertainty  appropriate  for  stressing  and  testing  effectively  the  DAMM  tools  relied  heavily  on  military 
judgement.  Generally,  vignette  complexity  was  matched  to  the  DAMM  capability  under  test.  White  Force  used 
more  complex  vignettes  and  injected  more  volatility  and  uncertainty  on  Objective  architecture  runs,  expecting 
better  mitigation  and  adaptation.  The  Baseline  architectures  were  tested  with  relatively  simpler  vignettes. 


A.5  DECISIONMAKING 

In  the  development  of  the  DAMM  CMDM  assessment  approach,  it  was  useful  to  consider  how  DAMM 
CMDM  task  MECs  were  structured  with  reference  to  existing  cognitive  frameworks.  Figure  A-2  illustrates  the 
structure  of  individual  CAS/TST  CMDM  task  MEC  examples  within  a  Skills-Rules-Knowledge  (SRK) 
cognitive  framework. 


DECISION  LEGEND 
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Figure  A-2:  DAMM  Decisions  in  the  SRK  Cognition  Framework. 
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The  SRK  framework  draws  distinctions  between  automatic  and  naturalistic  or  recognition-primed  decision 
making  decision  making,  and  deliberative,  analytical  and  evaluative  decision  making.  In  Figure  A-9,  the  nine 
CAS/TST  CMDM  task  MECs  are  shown  as  residing  at  the  rule-based  association  level,  and  at  the  knowledge- 
based  interpretation  and  evaluation  level. 

The  SRK  framework  concerns  cognition  at  the  level  of  the  individual.  DAMM  concerns  individuals  working 
collaboratively  within  a  distributed,  hierarchical  C2  process.  So,  it  was  also  considered  useful  to  examine  how 
DAMM  CMDM  task  MECs  were  structured  with  reference  to  C2  framework.  Cognitive  control  theory 
represents  cognition  as  a  layered  process  of  multiple  control  loops.  Figure  A-3  illustrates  a  representation  of 
the  structure  of  the  CAS/TST  CMDM  task  MEC  examples  within  the  Operational  and  Tactical  C2  architecture 
C2  OODA  (Observe>Orient>Decide>Act),  or  “COODA  loop”  layered  control  system.  Here,  the  REMDAER* 
framework  provides  the  components  for  multi-player,  distributed,  or  team,  decision  making  cycle. 


*Recognise>Evaluate>Mitigate>Disseminate>Acknowledge>Decide>Execute>Report. 
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Figure  A-3:  DAMM  Decisions  in  the  REMDAER  Framework. 


In  the  development  of  DAMM  capability,  and  in  planning  and  reporting  of  trials,  it  was  found  to  be  useful  to 
provide  system-of-systems  views  and  system  architecture  representations  of  DAMM  CMDM  derived  from 
MODAF/DODAF  system  architect  tools.  Figure  A-4  illustrates  a  representation  of  the  structure  of  decision 
making  using  a  systems  architecture  framework  view  approach  (MODAF/DODAF),  with  CAS/TST  CMDM 
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examples  depicted  in  sequential  order,  with  the  Command  Flow  across  the  architecture  tiers  (CAOC  White 
Force;  E3  OpTEAM;  TFJ  TacTEAM  and  TDSS),  and  with  the  Mission  Flow  within  Tiers.  This  approach  is 
more  suitable  for  identifying  CMDM  characteristics  such  as  influencing  factors,  prioritisation,  and  alternative 
Courses  of  Action  (CoA). 
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Figure  A-4:  DAMM  Decisions  in  Command  and  Mission  Flow  Framework. 
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In  DAMM  SE  trials  workshops,  the  impact  of  DAMM  on  mission  command  flow  was  frequently  discussed 
and  debated,  in  particular  the  increased  potential  for  distributed  adaptive  decision  making  providing  support 
for  the  role  of  Mission  Commander.  It  was  hypothesised  that  in  more  highly  networked  collaborative 
architectures,  SSA  might  be  more  widely  and  better  distributed,  and  that  mission  command  decision  making 
processes  might  not  necessarily  need  to  be  centralised,  as  illustrated  in  Figure  A-5.  As  reported  earlier,  in  the 
2nd  Joint  US-UK  SE  Trial,  September  2010,  four  different  MC  positions  were  tested,  and  the  Objective 
architecture  provided  relatively  good  adaptability  proficiency  with  the  MC  in  all  four  positions,  consistent 
with  good  communications  and  SA.  Individual  runs  showed  benefits  of  distributed  and  adaptive  decision 
making,  afforded  most  by  the  networked  Objective  architecture.  Further  work  is  needed  to  more  fully 
understand  the  implications  of  DAMM  for  Mission  Commander  MECs. 
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Figure  A-5:  Effect  of  DAMM  Network  Architecture  on  Mission  Command. 


A.6  DYNAMIC  MISSION  ACTIVITY  REPRESENTATION 

In  analysis  and  reporting  of  successive  trials,  the  need  was  recognised  to  develop  improved  methods  for  the 
representation  of  the  missions.  Mission  representations  needed  to  highlight  the  important  relationships 
between  system  components,  participants,  tasks,  goals,  events,  decisions  and  outcomes.  This  was  needed  in  a 
manner  that  captured  the  structure  of  the  dynamics  and  flow  and  afforded  measurement  of  performance. 
Illustrations  of  the  forms  of  representation  that  evolved  under  DAMM,  and  that  were  found  to  be  useful, 
is  shown  in  Figure  A-6  to  Figure  A- 8  below. 
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Figure  A-6:  DAMM  Mission  Command  Decision  Flow. 
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RUN  1  BASELINE  MC  E3 
AF2T2EA2  KILL  CHAIN 


1.  Kill/capture  MVI.  White 
van  from  docks  w/suspected 
medium  value  individual 
(MVI)  FDP  leader,  upgraded 
high  (HVI)  en-route.  Plannee 
ambush  at  meeting  point 
(RP)  nr  Invergarry  1  minute 
pre  meet,  storm  bldg  with 
SH  troops,  &  kill/capture 
leader  group. 

2.  Kill/capture  leader 
group.  Red  car  departing 
from  Laggan  docks  in 
convoy  with  white  van, 
with  accompanying  FDP 
enemy  forces  (EF). 
Kill/capture  leadergroup. 


3.  Destroy  Weapons.  Pre¬ 
planned  strike,  1  min  post 
RW  trp  drop,  against  anti¬ 
aircraft  DShKs  &  supplies 
(fuel,  ammo),  in  Abbey 
style  buildings,  suspected 
weapons  factory,  nr  south 
Ft  Augustus,  highly  likely 
to  be  taken  out  at  time  of 
meeting. 

4.  Destroy  C2  node.  C2 

building  near  village  of 
Invergarry  and  suspected 
MVI  convoy  meeting 
point. 


10:00:00 


10:10:00 


Figure  A-7:  DAMM  AF2T2EA  Kill  Chain. 
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RUN  1  BASELINE  MC  E3 

Mission  Event  Decision  Process 
Adaptability  Proficiency 


10:00:00  10:10:00  10:20:00 

m  1 1 1 1  n  1 1  in  1 1  ii  1 1 1 1 1 1 1 


10:30:00 

1  I  I  II 


10:40:00 

I  I  I  I  I  I  I  I  I 


"SH  abort;  AH  hard  stop " 

la  10:13:00  RP  Blow  through 

Mission  Objectives 

1.  Kill/capture  MVI. 

2.  Kill/capture  leader  group. 


DPI  Event:  Vehicle  does  not  stop  - 
Replan  decision 


14 


2.8 


DP2  Event:  Upgrade  MVTto  HVT-  RW 
Rerouting  and  role  (AH  attack  to 
coerce)  decision 


J 


11 


DP3  Event:  Effect  capture  in  new 
location-  Replan  capture -location 
and  tactics 


W  M 


J 


2.3 


W  =  White  Force  R  =  Rotary  wing 

M  =  Mission  commander  F  =  Fast  jet 

E=E3  i=  Total 

J  =  JTAC  Mn  =  Mean 


DP2WF3 

E3 

• 

MC 

• 

JTAC 

rwA  fj 

<DwO 

E3 

• 

MC 

• 

JTAC 

RW©  fj 

®  o 

E3 

O 

MC 

O 

JTAC 

RW©  FJ 

•  o 

"SH  abort;  AH  hard  stop 


10:05:19  10:09:40 
JTAC45  INT: 
TOI/PID  MVI>HVI  ' 
MVI  restriction * 


Replan- 
Track  HVI  vehicle 


Pre-planned  RW 
C2  RP  troop  drop 
for  MVI 
kill/capture 


VA  v  <8>V 


10:07:10 

JTAC42ID 

stationary 

van 


10:13:00 
JTAC45 
IDRP 
blow 
through ' 


T 


10:17:25 
MC  approved 
RWmove 
north/engage 
HVI  vehicle 


I 

fEaarra 

15 
RWAH 

strike/destroy 
HVI  vehicle 


^  10:26: 


10:15:25 
RWSH  abort 
ibv  troop  drop 
/HVI  capture 


^  Information 
Decision  Point 
Alternative  CoA 
^  Chosen  CoA 

A  Target 


Decision  00:04:25 
Action  00:08:45 

Response  00:13:15 
Change  ToT  00:11:55 


Figure  A-8:  DAMM  Adaptability  Proficiency  Decision  Mapping. 


A.7  TOOL  USAGE 

The  creation  of  scenarios  and  missions  that  properly  and  fully  exercised  the  envisaged  use  of  the  DAMM  tools 
proved  problematic.  This  arose  because  of  difficulties  in  effectively  mandating  DAMM  tool  use  during  the 
test  trials.  Variability  in  tool  usage  is  partly  a  training  issue.  However,  although  the  DAMM  tools  are  regarded 
as  enabling  technologies,  fundamentally  they  are  designed  to  provide  operator  aiding  and  decision  support. 
Tool  use  is  optional.  Simple  mission  management  tasks  can  be  completed  “manually”  without  the  aid  of 
DAMM  tools  (c.f.  Baseline  architecture),  relying  only  on  the  operator’s  airmanship  and  tactical  knowledge 
and  skills.  Whether  or  not  the  DAMM  tools  actually  get  used  in  a  realistic  trials  environment  is  dependent  on 
the  operator’s  training  and  perceptions  of  utility,  benefit,  and  ease  of  use,  as  judged  in  the  mission  context. 
Mitigation  of  the  risk  of  non-usage  of  tools  can  be  achieved  by  identification  of  strong  tools  use  cases,  and  by 
integration  of  validated  use  cases  into  the  trials  scenario  missions. 


A.8  LEVELS  OF  PROOF 

The  DAMM  programme  of  work  involved  progressive  development  and  test  with  increasing  levels  of  proof 
and  evidence  of  integration  de-risking  and  system  performance.  The  work  progressed  from  laboratory  bench 
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testing,  through  Synthetic  Environment  (SE)  trials,  to  Live,  Virtual  and  Constructive  (LVC)  environments  and 
flight  test.  The  levels  of  proof  and  weight  of  evidence  required  for  technology  demonstration  and  test  are 
associated  with  the  Technology  Readiness  Level  (TRL)  of  the  systems  under  test,  and  the  needs  for  risk 
reduction  and  cost/benefit  assurance.  For  progressing  development  of  concept  prototypes  at  relatively  low 
TRLs  1-4,  laboratory  bench  testing  and  SE  evaluations  of  mission  enabling  technologies  can  be  appropriate, 
using  only  core  sub-systems,  semi-realistic  missions  and  part-task  simulations,  comparing  only  essential 
equipment,  messages  and  links,  and  varying  critical  characteristics  of  the  operating  environment,  missions, 
stresses  and  tasks.  Here,  relatively  low  levels  of  evidence  of  performance  and  effectiveness  can  provide 
necessary  and  sufficient  for  proof  of  progress  and  assurance  of  concept  validity,  e.g.,  nominal/ordinal 
qualitative  data  level  metrics,  aircrew  subjective  ratings,  operator  usability  questionnaires.  At  high  TRLs  (5+), 
demonstrating  de-risking  and  readiness  for  exploitation  in  real  systems,  LVC  and  flight  test  of  mission 
enabling  technology  are  needed,  using  real  environments  and  stresses,  current  equipment  and  systems,  with 
objective  measurement  of  performance  and  effectiveness  on  realistic  operational  missions  and  tasks,  and 
demonstrations  of  real  effects. 

In  an  advanced  simulated  environment,  features  and  components  can  be  varied  up  to  high  levels  of  fidelity 
and  representativeness,  within  constraints  of  time  and  cost.  In  a  programme  with  progressive  test  and 
evaluation,  not  all  the  system  features  need  necessarily  simulated  at  a  uniformly  equivalent  level  of  fidelity, 
e.g.,  co-ordinated  aircraft  behaviours,  outside  world  visual  resolution,  sensors  and  communications  performance, 
C2  procedures,  cockpit/crew  workstation  layout,  HMI.  For  mission  systems  testing,  the  design  of  the  SE  test 
environment  representativeness  should  provide  the  standard  of  fidelity  necessary  and  sufficient  to  accomplish 
the  specific  test  objectives.  Higher  levels  of  SE  representativeness  should  be  needed  for  features  involved 
directly  in  the  performance  of  critical  mission  functions.  For  networked  critical  mission  system  functions  and 
associated  tasks,  interactions  and  procedures,  particular  consideration  needs  to  be  given  to  the  requirements 
for  representativeness  of  SA  and  tactical  information,  and  data  link  communication  of  tasks,  threats,  targets, 
tactics,  and  positions  of  other  assets,  routes  and  airspace. 

The  Operator-Mission  Interface  (OMI)  is  a  critical  component  of  DAMM.  Involvement  of  experienced 
military  operators  is  essential  at  all  the  levels  of  mission  system  development,  test  and  evaluation. 
Experienced  aircrew  are  needed  to  build  credible  and  realistic  test  scenarios  and  missions.  They  are  needed  to 
design  representative  stressing  missions  and  events  to  test  and  stress  the  mission  systems  and  aircrew  under 
evaluation.  Experienced  operators  are  needed  to  adapt  and  apply  realistic,  current  or  developmental  CONOPS, 
training,  Tactics,  Plans  and  Procedures  (TPP).  Mission  system  test  trials  need  scenarios  and  missions  to  focus 
on  crew  information  quality,  decision  making,  prioritization,  mission  command  and  interoperability  issues. 
Critically,  they  are  needed  to  provide  imagination,  creativity  and  expertise  to  develop  new  tests  for  new 
concepts  and  technologies,  where  for  DAMM  the  focus  is  on  the  efficiency  and  effectiveness  of  distributed 
adaptive  decision  making  in  a  highly  dynamic  networked  environment. 


A.9  MEASUREMENT  AND  METRICS 

Assessment  approaches  should  use  a  combination  of  objective  metrics  of  mission  performance,  and  operator 
provided  expert  judgments  captured  using  subjective  rating  scales.  Experience  has  shown  that  subjective  ratings 
of  aircrew  and  system  performance,  captured  using  simple,  reliable  and  proven  methods,  provides  valuable 
quantitative  evidence  and  insight  on  decision  making  performance,  that  aids  and  reinforces  the  interpretation  of 
objective  data.  Metrics  of  should  include  ratings  of  decision  quality,  specifically  survivability,  effectiveness  and 
timeliness,  in  addition  to  SA  and  workload. 
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The  DAMM  programme  sought  to  develop  a  sensible  and  practical  set  of  simple  rating  scale  protocols  for  data 
capture,  during  both  real-time  on-line  assessments  by  SME  Observers,  and  from  crew  participants  during  post¬ 
run  de-briefings.  Several  versions  of  the  basic  structure  were  employed  across  the  trails.  Item  content  was  varied 
and  refined  following  feedback  from  users  and  statistical  evidence  of  item  sensitivity  and  discriminative  power. 
The  evidence  indicated  the  value  of  Team  Work  metrics,  in  addition  to  measurement  of  individual  Task  Work, 
for  measuring  operator  performance  in  distributed,  collaborative  networked  operations.  The  protocols  used 
towards  the  end  of  the  DAMM  programme,  and  the  associated  metrics  structure,  are  shown  in  Figure  A-9  to 
Figure  A- 12  below. 
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PARTICIPANT  PROTOCOL  RATING  SCALE  INTERPRETATION 


LOW  ANCHORS 

DIMENSION 

HIGH  ANCHORS 

Lowl  2  3  4  5  6  7High 

WORKLOAD 

Slow/Leisurely/No  time  pressure 

WLTime  Pressure 

Rushed/Rapid/Frantic 

Very  little/Extremely  easy 

WL  Mental  Effort 

Extremely  hard/Stretched 

Relaxed/Placid/T ranquil/Calm 

WL  Stress 

Anxious/Worried/Up  tight/ Harassed 

Light/Quiet/Unpressured 

Team  Workload 

Heavy/Busy/Pressured 

REPLAN  TASK  LOAD 

Forgiving/Simple/Easy 

Replan  Decision 

Recognise-Evaluate-Mitigate 

Difficult/ Demanding/ Complex/ Exacting 

Forgiving/Simple/Easy 

Replan  Action 

Disseminate-Acknowledee-Execute-ReDort 

Difficult/Demanding/Complex/Fxacting 

SITUATION  AWARENESS 

Stable/Simple/Fixed 

SA  Demand  on  Attentional  Resources 

Unstable/Complex/Variable 

Relaxed/lnattentive/Spare  Capacity/Singular 

SA  Supply  of  Attentional  Resources 

Alert/Concentrated/Full  capacity/Divided 

Uninformative/Meaningless/Unfamiliar 

SA  Understanding 

Informative/Meaningful/Familiar 

DECISION  QUALITY 

Indecisive/  Doubtful/Guess 

DQ  Confidence 

Decisive  /Confident  /Obvious 

Dangerous/Vulnerable/Risky 

DQ  Survivability 

Safe/Unthreatening/Secure 

Ineffective/Un-useful/Unproductive 

DQ  Effectiveness 

Effective/Capable/Useful 

Behind/Late/Pressurised 

DQ  Timeliness 

Ahead/On-time/In  control 

PERFORMANCE 

Unsatisfactory/Unacceptable/Failure 

Task  Performance 

Perfect/Successful/Satisfied 

Useless/Complicated/Difficult 

Tools  Utility 

Easy/Intuitive/Effective 

No  backup  plans/  Insensitive/Unresponsive 

Adaptability  Proficiency 

Sensitive/Responsive/Timely  effective  backup  plans 

Fails  to  meet  any  mission  objectives/0% 

Probability  of  Mission  Success 

100%/Successfully  achieved  all  mission  objectives 

Figure  A-9:  DAMM  Participant  CMDM  Assessment  Protocol. 
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OBSERVER  PROTOCOL  RATING  SCALE  INTERPRETATION 


LOW  ANCHORS 

DIMENSION 

HIGH  ANCHORS 

Lowl  2  3  4  5  6  7High 

DECISION  QUALITY 

Dangerous/Vulnerable/Risky 

DQ  Survivability 

Safe/Unthreatening/Secure 

Ineffective/Useless/Unproductive 

DQ  Effectiveness 

Effective/ Capable/ Useful 

Behind/Late/Pressurised 

DQ  Timeliness 

Ahead/On-time/In  control 

TEAMWORK 

Silent/Unclear/Confused/Slow/Uninformative 

Communication 

Clear/Timely/Coherent/Concise/Rapid/Informative 

Uninformative/Meaningless/Unfamiliar 

Shared  SA 

Informative/ Meaningful/ Familiar 

Indecisive/ Unassertive/Confused/ Uncommunicative 

Leadership 

Commanding/Directing/Gui  din  g/lniti  ating 

Ignoring/ Hindering/ Blocking/Criticisin  g/Conflictin  g 

Support 

Monitoring/Advising/Correcting/Assisting 

Ljght/Quiet/Unpressured 

Team  Workload 

Heavy/Busy/Pressured 

PERFORMANCE 

Unsatisfactory/Unacceptable/Failure 

Task  Performance 

Perfect/Successful/Satisfied 

Separate/Independent/Divided/Conflicting 

Collaboration 

Joint/Shared/Coordinated/Cooperating 

Powerless/Weak/Irrelevant/Not  Involved 

Influence  Power 

Influential/Powerful/Forceful/Decisive/Critical 

No  backup  plans/  Insensitive/Unresponsive 

Adaptability  Proficiency 

Sensitive/Responsive/Timely  effective  backup  plans 

Fails  to  achieve  any  mission  objective/0% 

Probability  of  Mission  Success 

100%/ All  mission  objectives  successfully  achieved 

Figure  A-10:  DAMM  Observer  Assessment  Protocol. 
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Figure  A-11:  Reward/Effort  Metrics  Structure. 
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Figure  A-12:  Collaboration  Metrics  Structure. 
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